Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
I am wondering what I should do for the purpose of my project.
I am gonna operate on about 100 000 rows, every time.
what I wanted to do is to create an object "{}" and then, if I need to search for a value, just call it , for example
data['2018']['09']['Marketing']['AccountName']
the second option is to pull everyting into an array "[]" and in case I need to pull value, I will create a function to go through the array and sum numbers for specific parameters.
But don't know which method is faster.
Will be thankful if you can shed some light on this
Thanks in advance,
If performance (speed) is an issue, Python might not be the ideal choice...
Otherwise:
Might I suggest the use of a proper database, such as SQLLite (which comes shipped with Python).
And maybe SQLAlchemy as an abstraction layer. (https://docs.sqlalchemy.org/en/latest/orm/tutorial.html)
After all, they were made exactly for this kind of tasks.
If that seems overkill: Have a look at Pandas.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 1 year ago.
Improve this question
I'm new to python, and I'm a bit confused about the use cases of data types in python
Can someone please explain in detail when to use each data type, with an example if possible
Thank you.
Lists are used when you have data you want to further modify, alter like sorting and all.
Dictionary is used when you have to sets of data where data of the first set corresponds to data of other set. And the position of the data doesn't matter only the relation of the two sets matters.
A tuple is used when position of the data is very important and you don't want to alter the position throughout.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 2 years ago.
Improve this question
If code is divided into too many segments, can this make the program slow?
For example - Creating a separate file for just a single function.
In my case, I'm using Python, and suppose there are two functions that I need in the main.py file. If I placed them in different files (just containing the function).
(Suppose) Also, If I'm using the same library for the two functions and I've divided the functions into separate files.
How can this affect efficiency? (Machine performance-wise and Team-wise).
It depends on the language, the framework you use etc. However, dividing the code too much can make it unreadable, which is (most of the time) the bigger problem. Since most of the time you will (or should) be working in a team, you should consider how readable your code would be for them.
However, answering this in a definite way is difficult. You should ask a Senior developer on your team for guidelines.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 3 years ago.
Improve this question
Often you see question asked about a better method of doing something, or just generally a looping question and very often the top answers will use some form of convoluted list/dict/tuple comprehension that takes longer for others to understand than create themselves. While a simple and understandable loop could have just been made.
Since it cannot provide any speed benefits that I could imagine, is there any use of it in python other than to look smart or be Pythonic?
Thanks.
I believe the goal in this case to make your code as concise and efficient as possible. At times it can seem convoluted, but the computer looping through multiple lines as opposed to a single line adds processing time, which in large applications and across many iterations can cause some delays.
Additionally, although it seems harder to understand initially, for an outside individual reading your code, it's much quicker for them to read simplified expressions than pages of loops to get an idea of what you're attempting to accomplish.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
Simply put, what are the preferred practices for writing larger python applications that use pandas dataframes as its primary method for data representation?
I often find myself struggling to maintain inconsistencies in dataframes, sometimes invariants leak through in data, datatypes are not what you expect etc.
I'm wondering just what are the best practices for writing larger, stable applications in pandas? I want to take advantage of array-representation in data for speed, but I also want to make sure that there's a way to further define the "bounds" of dataframe, what it should have in it, in a clean way.
Assertions on receiving a dataframe from a caller.
Forcing a dataframe parameter to have specific dtypes.
Defining a dataframe "type" based upon the columns it has.
Opportunities for OOP, at the dataframe level
Also, sorry for the vague nature of this. I'm starting on a project, and I want to ask this question before I get too far off course. I've been burned in the past with regards to not enforcing enough of a structure when it comes to dataframes.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I need to create and search a big list, for example list of 3.9594086612e+65 arrays of 10x10 (or bigger list of bigger arrays).
I need to create list of combinations. After I need to filter out some combinations according to some rule. This process should be repeated till only one combination is left.
If I try to create this list it crashes because of the memory after few minutes.
I can imagine, that solution should be to store the data in different way than list in memory.
What is possible, correct and easy way? SQL database? NoSQL database? Or just multiple text files opened and closed one after one? I need to run through this list multiple times.
Hy:
3.9594086612e+65
It's more than all computer memory in the world.
(And I do not multiply by 10)!