We Keep Coding

How to open .ndjson file in Python?

I have .ndjson file that has 20GB that I want to open with Python. File is to big so I found a way to split it into 50 peaces with one online tool. This is the tool: https://pinetools.com/split-files
Now I get one file, that has extension .ndjson.000 (and I do not know what is that)
I'm trying to open it as json or as a csv file, to read it in pandas but it does not work.
Do you have any idea how to solve this?
import json
import pandas as pd
First approach:
df = pd.read_json('dump.ndjson.000', lines=True)
Error: ValueError: Unmatched ''"' when when decoding 'string'
Second approach:
with open('dump.ndjson.000', 'r') as f:
my_data = f.read()
print(my_data)
Error: json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 104925061 (char 104925060)
I think the problem is that I have some emojis in my file, so I do not know how to encode them?

ndjson is now supported out of the box with argument lines=True
import pandas as pd
df = pd.read_json('/path/to/records.ndjson', lines=True)
df.to_json('/path/to/export.ndjson', lines=True)

I think the pandas.read_json cannot handle ndjson correctly.
According to this issue you can do sth. like this to read it.
import ujson as json
import pandas as pd
records = map(json.loads, open('/path/to/records.ndjson'))
df = pd.DataFrame.from_records(records)
P.S: All credits for this code go to KristianHolsheimer from the Github Issue

The ndjson (newline delimited) json is a json-lines format, that is, each line is a json. It is ideal for a dataset lacking rigid structure ('non-sql') where the file size is large enough to warrant multiple files.
You can use pandas:
import pandas as pd
data = pd.read_json('dump.ndjson.000', lines=True)
In case your json strings do not contain newlines, you can alternatively use:
import json
with open("dump.ndjson.000") as f:
data = [json.loads(l) for l in f.readlines()]

Reading XLSB (binary) file with Pandas read_excel using pyxlsb reads empty rows for some xlsb file

I'm trying to read binary Excel files using read_excel method in pandas with pyxlsb engine as below:
import pandas as pd
df = pd.read_excel('test.xlsb', engine='pyxlsb')
If the xlsb file is like this file (Right now, I'm sharing this file via WeTransfer, but if there is a better way to share files on StackOverflow, let me know), the returned dataframe is filled with NaN's. I suspected that it might be because the file was saved with active cell pointing at the empty cells after the data originally. So I tried this:
import pandas as pd
with open('test.xlsb', 'rb') as data:
data.seek(0,0)
df = pd.read_excel(data, engine='pyxlsb')
but it still doesn't seem to work. I also tried reading the data from byte number 0 (from the beginning), writing it into a new file, 'test_1.xlsb', and finally reading it with pandas, but that doesn't work.
with open('test.xlsb','rb') as data:
data.seek(0,0)
with open('test_1.xlsb','wb') as outfile:
outfile.write(data.read())
df = pd.read_excel('test_1.xlsb', engine='pyxlsb')
If anyone has suggestion as to what might be going on and how to resolve it, I'd greatly appreciate the help.

Encoding issues when reading from CSV via pandas.read_csv

I've exported a CSV-file from MetaTrader5 via MQL5 expert advisor.
EURUSD,2020.02.19 05:04:00,1.07991,1.07991
EURUSD,2020.02.19 05:05:00,1.07991,1.07989
EURUSD,2020.02.19 05:06:00,1.07989,1.07988
EURUSD,2020.02.19 05:07:00,1.07988,1.07989
EURUSD,2020.02.19 05:08:00,1.07989,1.0799
...
Now I need to read this CSV-file with Pandas. When I use the following code...
import pandas as pd
df_rates = pd.read_csv('D:/Rates.csv', header=None, encoding='cp1252')
df_rates.columns = ['Currency','Time','Open','Close']
print(df_rates)
I see 'NaN' instead of all of my data. I tried different encoding settings but this doesn't help. I have operating system with cyrillic settings. Any suggestions?
Exported CSV-file is here.

Pandas: No columns to parse from file

Good afternoon
I have looked through several of the solutions linked to this problem and nothing has been able to help me. I do not understand if it is an error with the actual csv file or an error within the code itself. Below is my code:
import pandas as pd
from itertools import islice
import csv
from cStringIO import StringIO
sio = StringIO()
def forex_file():
with open("USD-ZAR.csv", "r+") as exchange_file:
for row in islice(csv.reader(exchange_file), 3, 256, None):
sio.write(row)
sio.seek(0)
df1 = pd.read_csv(sio, sep=",", encoding="utf-8", delim_whitespace=True)
I purposely placed the "delim_whitespace=True" part within the code as this has been the common suggestion in many of the other posts but is has done nothing in this case as my csv file is split by normal commas and not by white space or tabs.
Any help will really be appreciated!

how to read a data file including "pandas.core.frame, numpy.core.multiarray"

I met a DF file which is encoded in binary format. But when I open it using Vim, still I can see characters like "pandas.core.frame", "numpy.core.multiarray". So I guess it is related with Python. However I know little about the Python language. Though I have tried using pandas and numpy modules, I failed to read the file. Could you guys give any suggestion on this issue? Thank you in advance. Here is the Dropbox link to the DF file: https://www.dropbox.com/s/b22lez3xysvzj7q/flux.df

Looks like DataFrame stored with pickle, use read_pickle() to read it:
import pandas as pd
df = pd.read_pickle('flux.df')