We Keep Coding

Reading survey data CSV with multiple selection sub-columns?

I would like to import this data from a Navigraph survey results.
https://navigraph.com/blog/survey2022
The dataset is here:
https://download.navigraph.com/docs/flightsim-community-survey-by-navigraph-2022-data.zip
However, I noticed the structure is something I'm not quite used to, and perhaps this is how a lot of polling data is shared. The semicolons being separators is not an issue. It's the fact there's a mix of "select multiple" responses as columns. The tidiest thing is starting at the third row, each row is a single respondent.
How can I clean up this data so it is as "tidy" as possible? How would I melt() these columns into rows? How do I handle the multiple selection responses in the sub-columns?
I'd like the questions and responses to simply be two columns respectively.

Hello how are you? I don't have full knowledge in this type of work but I believe you will have to:
1- Read the file as is
2- Concatenate the columns of questions and answers
3- Create the dataset that will be used
I believe that pandas has some commands that will help you, just find the patterns to define what are "questions" and "answers" in this dataset.

Read tabular data (rows and named columns) in Python - best practices

What is the best way to read data from txt/csv file, separate values based on columns to arrays (no matter how many columns there are) and how skip for example first row if file looks like this:
Considering existing libraries in python.
So far, I've done it this way:
pareto_front_file = open("Pareto Front.txt")
data_pareto_front = pareto_front_file.readlines()
for pareto_front_row in data_pareto_front:
x_pareto.append(float(pareto_front_row.split(' ')[0]))
y_pareto.append(float(pareto_front_row.split(' ')[1]))
but creating more complicated things I see that this way is not very effective

Use the "Pandas" library (or something similar)
For tabular data, one of the most popular libraries is Pandas. Not only will this allow you to read the data easily, there are also methods for nearly all types of data transformation, filtering, visualization, etc. you can imagine.
Pandas is one of the most popular python packages, and although it may seem daunting at first, it is usually a lot easier than re-inventing the wheel yourself.
In case you are familiar with the R language, pandas covers a lot of the tidyverse functionalities and the DataFrame is similar to R's data.frame.
Transfer your data into a python object
It offers a read_csv method where you can specify a custom delimiter if you need it. Your data will come as a pandas "DataFrame", a datatype specifically designed for the kind of data you describe. Among many other things, it will recognize column names such as the ones you have in your data automatically.
Example for csv:
df = pd.read_csv('file.csv', delimiter=',') # choose delimiter you need
print(df.head()) # show first 5 rows
print(df.summary()) # get an overview over your data

Python - What are the major improvement of Pandas over Numpy/Scipy

I have been using numpy/scipy for data analysis. I recently started to learn Pandas.
I have gone through a few tutorials and I am trying to understand what are the major improvement of Pandas over Numpy/Scipy.
It seems to me that the key idea of Pandas is to wrap up different numpy arrays in a Data Frame, with some utility functions around it.
Is there something revolutionary about Pandas that I just stupidly missed?

Pandas is not particularly revolutionary and does use the NumPy and SciPy ecosystem to accomplish it's goals along with some key Cython code. It can be seen as a simpler API to the functionality with the addition of key utilities like joins and simpler group-by capability that are particularly useful for people with Table-like data or time-series. But, while not revolutionary, Pandas does have key benefits.
For a while I had also perceived Pandas as just utilities on top of NumPy for those who liked the DataFrame interface. However, I now see Pandas as providing these key features (this is not comprehensive):
Array of Structures (independent-storage of disparate types instead of the contiguous storage of structured arrays in NumPy) --- this will allow faster processing in many cases.
Simpler interfaces to common operations (file-loading, plotting, selection, and joining / aligning data) make it easy to do a lot of work in little code.
Index arrays which mean that operations are always aligned instead of having to keep track of alignment yourself.
Split-Apply-Combine is a powerful way of thinking about and implementing data-processing
However, there are downsides to Pandas:
Pandas is basically a user-interface library and not particularly suited for writing library code. The "automatic" features can lull you into repeatedly using them even when you don't need to and slowing down code that gets called over and over again.
Pandas typically takes up more memory as it is generous with the creation of object arrays to solve otherwise sticky problems of things like string handling.
If your use-case is outside the realm of what Pandas was designed to do, it gets clunky quickly. But, within the realms of what it was designed to do, Pandas is powerful and easy to use for quick data analysis.

I feel like characterising Pandas as "improving on" Numpy/SciPy misses much of the point. Numpy/Scipy are quite focussed on efficient numeric calculation and solving numeric problems of the sort that scientists and engineers often solve. If your problem starts out with formulae and involves numerical solution from there, you're probably good with those two.
Pandas is much more aligned with problems that start with data stored in files or databases and which contain strings as well as numbers. Consider the problem of reading data from a database query. In Pandas, you can read_sql_query directly and have a usable version of the data in one line. There is no equivalent functionality in Numpy/SciPy.
For data featuring strings or discrete rather than continuous data, there is no equivalent to the groupby capability, or the database-like joining of tables on matching values.
For time series, there is the massive benefit of handling time series data using a datetime index, which allows you to resample smoothly to different intervals, fill in values and plot your series incredibly easily.
Since many of my problems start their lives in spreadsheets, I am also very grateful for the relatively transparent handling of Excel files in both .xls and .xlsx formats with a uniform interface.
There is also a greater ecosystem, with packages like seaborn enabling more fluent statistical analysis and model fitting than is possible with the base numpy/scipy stuff.

A main point is that it introduces new data structures like dataframes, panels etc. and has good interfaces to other structure and libs. So in generally its more an great extension to the python ecosystem than an improvement over other libs. For me its a great tool among others like numpy, bcolz. Often i use it to reshape my data, get an overview before starting to do data mining etc.

How can I create a formatted and annotated excel with embedded pandas DataFrames

I want to create a "presentation ready" excel document with embedded pandas DataFrames and additional data and formatting
A typical document will include some titles and meta data, several Data Frames with sum row\column for each data frame.
The DataFrame itself should be formatted
The best thing I found was this which explains how to use pandas with XlsxWriter.
The main problem is that there's no apparent method to get the exact location of the embedded DataFrame to add the summary row below (the shape of the DataFrame is a good estimate, but it might no be exact when rendering complex DataFrames.
If there's a solution that relies on some kind of template, and not hard coding it would be even better.

How to store multiple related time series in Pandas

I'm new to Pandas and would like some insight from the pros. I need to perform various statistical analyses (multiple regression, correlation etc) on >30 time series of financial securities' daily Open, High, Low, Close prices. Each series has 500-1500 days of data. As each analysis looks at multiple securities, I'm wondering if it's preferable from an ease of use and efficiency perspective to store each time series in a separate df, each with date as the index, or to merge them all into a single df with a single date index, which would effectively be a 3d df. If the latter, any recommendations on how to structure it?
Any thoughts much appreciated.
PS. I'm working my way up to working with intraday data across multiple timezones but that's a bit much for my first pandas project; this is a first step in that direction.

since you're only dealing with OHLC, it's not that much data to process, so that's good.
for these types of things i usually use a multiindex (http://pandas.pydata.org/pandas-docs/stable/indexing.html) with symbol as the first level and date as the second. then you can have just the columns OHLC and you're all set.
to access multiindex use the .xs function.

Unless you are going to correlate everything with everything, my suggestion is to put this into separate dataframes and put them all in a dictionary, ie {"Timeseries1":df1, "Timeseries 2":df2...}. Then, when you want to correlate some timeseries together, you can merge them and put suffixes in the columns of every different df to differentiate between them.
Probably you are interested in this talk Python for Financial Data Analysis with pandas by the author of pandas himself.