We Keep Coding

Calling python functions / files from PHP - request.Session - Login/Logout functionality

I am working on my scrapping project which requires login to the page, and two questions just appeared in front of me. Ok, maybe two blocks of questions. (please have mercy with me as I am beginner learning new things and probably I already poorly designed the whole project).
The design is: as the page requires login, I wanted to use python request.Session() method object and with "with context manager" keep to be logged in.
So my first question is:
1, If I want to have, lets say main.php file, which will call python file with login function (the one with context manager), do I have to have the rest of the functions in this same python file (all the functions for the specific scraping) ? more precisely all functions within this "with block" ? What if I want to have each important function in separate python file (one for login.py, another update.py, table.py, etc)? If I call other functions than login function, how the session will be still valid and I wont get logged out ? or should I just call the functions with "s" object somehow ?
Second question:
2, It's related due the sessions. I went through some questions and not sure if I found the correct answer. But when I logged into the page, through network tab on chrome I saw some session ID. But when I checked session ID via "what_in_session = s.cookies.get_dict()",
ID was different. When I moved on the page the session id in network was still same, however, via python was different every time.
Am I doing something wrong ? Or is it correct behavior ? I found this but I am not sure: Why are the IDs different in each request?.
I guess, once the "with request.Session" block is used, I dont need to care about the rest ? Then back to the first question, how to mange these python functions ?
Hope that you'll understand the questions. (if not I can try to rephrase them)
many thanks

Test Driven Development with Django

I have a conceptual question about doing test driven development with Django, may also apply to other frameworks as well.
TDD states that the firts step in the development cycle is to write failing tests.
Suppose for a unit test, I want to verify that an item is actually created when a request arrives. To test this functioanlty, I want to issue a request with the test client, and check with the db that this object is actully created. To be able to do that, I need to import the related model in the test file, but as the first step is writing this test, I don't even have a model yet. So I won't be able to run the tests to see them fail.
What is the suggested approach here? Maybe write a simpler test first, then modify the test after enough level of production code is implemented?

Important note: What you describe is not a unit test. It does not test one unit. It tests whole bunch of things starting form django url wiring, views and ending with models. This is integration tests. Secondly, don't create a test that uses external API (or test client which is mostly the same) to create data but checks that entity was created by going directly to DB. This is not good. If you create data via some API you should use the API of the same level to check the data is created. So my explanation will talk about this approach.
What you describe is a common problem when you start with TDD.
Important things about TDD are that you:
do small steps
refactor after test is green (including test refactoring)
This may sound like simple and you most probably have read and know that but the consequences for how you structure you work might not be that obvious.
Main consequence is that you do not write the full test from the scratch before implementing the functionality. You start with simplest test you can do, make it work (by implementing some piece of functionality), refactor. Then you change the test by adding more things that you want to check to it, implement that piece to make test green, refactor and so on.
That has consequence that you need to split the work (or plan how you implement it by simple steps) to be able to work in this mode. This requires some practice and I guess is one the main barriers for TDD adoption.
It is similar (but with important difference) to what you wrote:
Maybe write a simpler test first, then modify the test after enough level of production code is implemented?
You need to have simple test first, then modify it iteratively with small steps but before you implement production code not after.
In this particular case you can implement it in following steps:
1 Create the test that uses test client
def test_entity_creation(self):
post_result = test_client.post(POST_URL, {})
get_result = test_client.get(get_entity_url_from(post_result))
assert_that(get_result, not_none())
You have a test that fails but no line of code is written.
Note that no data is passed yet and the check is very basic.
2 Create url wiring and empty view
Do it so that the test pass. You need very few changes in the code and the view will not return much if anything. View can return some hardcoded json/dict at this point.
3.1 Check that id of the entity is generated
def test_entity_creation(self):
post_result = test_client.post(POST_URL, {})
get_result = test_client.get(get_entity_url_from(post_result))
assert_that(get_result, not_none())
assert_that(get_result, has_field('id', not_none()))
You can make this test work by adding id to the hardcoded dict.
3.1 Check that unique id of the entity is generated
Add a new test that checks that ids are unique:
def test_create_generates_unique_id(self):
post_result1 = test_client.post(POST_URL, {})
post_result2 = test_client.post(POST_URL, {})
assert_that(get_id(post_result1), not_(equal_to(get_id(post_result2)))
4 Add the model with only id
It is not hard to add a model with only id and add its creation and retrieval from the view. Don't add all the fields that you need, you will do that step by step later.
5 Add one field to you test
def test_entity_creation(self):
post_result = test_client.post(POST_URL, {'field': 'value'})
get_result = test_client.get(get_entity_url_from(post_result))
assert_that(get_result, not_none())
assert_that(get_result, has_field('field', 'value'))
Add a field to the model and make the test pass.
6 Continue doing TDD
Add more tests and production code.
Some more thoughts
Step 4 might be too big as for one TDD cycle. It requires making changes at least to three things:
post view handler
get view handler
model
In many cases it makes sense to split it by first creation a test for the model itself. The test that will not work with test client but will look like this:
def test_entity(self):
entity = Entity.objects.create()
entity = Entity.objects.get(entity.id)
assert_that(entity.id, not_none())
Then you add a model. Make sure that test_entity pass and only after that modify view to use you (already tested) model.
I hope this gives the idea how to approach this problem.

In Django, the approach is always to recreate a working environment for testing and staging. In testing the data is fake, in staging the data is "old" or very similar to the production.

Ambiguous step in Python Behave

My business user likes to use the then sentence "It should be created", where it is determined by the context of the scenario. For example:
Given I have gift certificate for "<name>"
When I enter the gift certificate
Then It should be created
or
Given Customer order return for order "<order_no>"
When I create the customer order return
Then It should be created
In the "Then It should be created", I would like to retrieve either the created gift certificate or customer order return for comparison. However, they have completely different API and object.
Firstly, is there a way to do this in Python Behave without getting "Exception AmbiguousStep:"?
If not, what would be best practice in BDD world for this without forcing user to have to repeat themselves constantly by saying "Then The gift certificate should be created" or "Then The customer order return should be created"?
Thanks.

In the specific case you are giving us here, I would write the steps more verbosely to avoid the "it". So I would write "Then the gift certificate should be created", etc. I prefer to avoid having steps depend on state passed through context.
However...
There are times where it would be problematic to do this. In your case, maybe the politics of dealing with your business user make it so that asking for more verbosity would not fly well. Or there can be technical reasons that cause what I suggested above to be undesirable or flat out unworkable.
What you can do if you cannot use more verbose steps, is have the Then it should be created step be dependent on a context field being set to a value that will provide enough information to the step to perform its work. It could be something like context.created_object. The step that creates the object would set this field to an appropriate value so that Then it should be created can perform its work. What exactly you would store in there depends on the specifics of your application.
For one application of mine where I test the appearance of a contextual menu on the basis of mouse clicks in a browser window, sometimes what I record is a reference to the DOM element on which the user did the right-click that brought up the menu. Sometimes it is an object providing x, y coordinates. This is what my application needs in order to perform its checks. In this case it is preferable to have the information be passed through context because having Selenium query the DOM all over again in later steps can be very expensive over the network. Over dozens of tests, it can easily add minutes to a test suite's run, and then consider that the suite has to be run for multiple combinations of browser, OS, and browser version.

Define the order of scenarios (or required scenario) with behave (python)

I'm using behave to test my little Django app.
I've already created the file user_management.feature that contains also this Scenario:
Scenario: register
Given I can access registration form
When I put "doctor" in "username" field
And I put "tardisBlue" in "password" field
And I put "doctor#tardis.com" in "email" field
And I press the "Register" button
Then the registration is successful
And I am logged in
Everythig works fine.
The next feature I want to develop is in file project_management.feature:
Scenario: create a project
Given I am logged in
When I go to the home page
And I click on "Create new Project" link
And I fill the fields
| field | text |
| name | Save Gallifrey |
And I click on "Save" button
And I go to the home page
Then I see the project name in the project list
Now when I execute my test, behave executes the feature files in alphabetical order, so project_management.feature is executed first.
It raise an error in the first given, because the user has not been created yet.
I've tested renamin the first file in 01_user_management.feature to make it work.
Do you know a better solution?
Is there some configuration file where I can specify the order of the feature file?
Or can I tell that a Scenario needs another Scenario tu run first?

You should not make scenarios dependent on one another. It is absolutely possible to do this. I have multiple large and complex test suites with hundred of scenarios. No scenario of mine depends on another scenario having run before it.
When you have a large suite and there's a single scenario failing, it is extremely useful to be able to do:
behave -n 'failing scenario name'
This gets Behave to run only the failing scenario. Alternatively, there's the #wip tag that can do the same thing. However, if the scenario you want to test depends on another one, Behave won't automatically know that it should run the other scenario so you have a) to know the dependency and b) manually take care of selecting all scenarios that the one you really want to run depends on.
What I'd do in your situation (which is pretty much what I've done in the past) is implement a step Given I am logged in as .... I implement it with a regex so that I can use
Given I am logged in as an administrator
Given I am logged in as a regular user
Given I am logged in as a user with permissions to delete articles
The application I'm testing has its database preloaded with some test users that correspond to the cases above. (There's still a test for registering new users but that's independent from the preloaded users.) The Given I am logged in as ... step just logs the user in. It does not need to create the user.
One of the side benefits of doing this is that if you run your suite on a test service like Sauce Labs or BrowserStack and use Selenium, you can implement the Given I am logged in as ... step to save a lot of testing time. Each Selenium command in such a case requires a round-trip between your Behave test and the browser running on the test service, which can take significant time transiting through the Internet. Reducing the number of such interactions can make a huge difference in the time it takes to run a whole suite.

There seems to be two ways to do this. One is that you can use Background to setup state for multiple Scenarios. The other is to call previous steps from other steps. The first solution would look something like this:
Feature: logins
Test login functionality
Background: login
Given I can access registration form
And I put "doctor" in "username" field
And I put "tardisBlue" in "password" field
And I put "doctor#tardis.com" in "email" field
And I press the "Register" button
Scenario: successful login
Then the registration is successful
And I am logged in
Scenario: create a project
When I go to the home page
And I click on "Create new Project" link
And I fill the fields
| field | text |
| name | Save Gallifrey |
And I click on "Save" button
And I go to the home page
Then I see the project name in the project list

Use feature-listfiles, like:
behave #my_foo.featureset …
with:
# -- FILE: my_foo.featureset
features/alice.feature
features/bob.feature:10

Answers from #Andrew Johnson and jenisys provides three ways to specify the order of steps/scenarios/features. And here I add a couple of more:
Basically same idea with listfile as jenisys mentioned, when specifying feature files in command line, features will be executed in the order as written. It seems not officially documented but this is what I observed.
One can also use tags to control what scenarios to run in command line. In the example of --tags=t1,t2,t3, those scenarios will be executed exactly the order written there. Handy.

How to setup a 3-tier web application project

EDIT:
I have added [MVC] and [design-patterns] tags to expand the audience for this question as it is more of a generic programming question than something that has direclty to do with Python or SQLalchemy. It applies to all applications with business logic and an ORM.
The basic question is if it is better to keep business logic in separate modules, or to add it to the classes that our ORM provides:
We have a flask/sqlalchemy project for which we have to setup a structure to work in. There are two valid opinions on how to set things up, and before the project really starts taking off we would like to make our minds up on one of them.
If any of you could give us some insights on which of the two would make more sense and why, and what the advantages/disadvantages would be, it would be greatly appreciated.
My example is an HTML letter that needs to be sent in bulk and/or displayed to a single user. The letter can have sections that display an invoice and/or a list of articles for the user it is addressed to.
Method 1:
Split the code into 3 tiers - 1st tier: web interface, 2nd tier: processing of the letter, 3rd tier: the models from the ORM (sqlalchemy).
The website will call a server side method in a class in the 2nd tier, the 2nd tier will loop through the users that need to get this letter and it will have internal methods that generate the HTML and replace some generic fields in the letter, with information for the current user. It also has internal methods to generate an invoice or a list of articles to be placed in the letter.
In this method, the 3rd tier is only used for fetching data from the database and perhaps some database related logic like generating a full name from a users' first name and last name. The 2nd tier performs most of the work.
Method 2:
Split the code into the same three tiers, but only perform the loop through the collection of users in the 2nd tier.
The methods for generating HTML, invoices and lists of articles are all added as methods to the model definitions in tier 3 that the ORM provides. The 2nd tier performs the loop, but the actual functionality is enclosed in the model classes in the 3rd tier.
We concluded that both methods could work, and both have pros and cons:
Method 1:
separates business logic completely from database access
prevents that importing an ORM model also imports a lot of methods/functionality that we might not need, also keeps the code for the model classes more compact.
might be easier to use when mocking out ORM models for testing
Method 2:
seems to be in line with the way Django does things in Python
allows simple access to methods: when a model instance is present, any function it
performs can be immediately called. (in my example: when I have a letter-instance available, I can directly call a method on it that generates the HTML for that letter)
you can pass instances around, having all appropriate methods at hand.

Normally, you use the MVC pattern for this kind of stuff, but most web frameworks in python have dropped the "Controller" part for since they believe that it is an unnecessary component. In my development I have realized, that this is somewhat true: I can live without it. That would leave you with two layers: The view and the model.
The question is where to put business logic now. In a practical sense, there are two ways of doing this, at least two ways in which I am confrontet with where to put logic:
Create special internal view methods that handle logic, that might be needed in more than one view, e.g. _process_list_data
Create functions that are related to a model, but not directly tied to a single instance inside a corresponding model module, e.g. check_login.
To elaborate: I use the first one for strictly display-related methods, i.e. they are somehow concerned with processing data for displaying purposes. My above example, _process_list_data lives inside a view class (which groups methods by purpose), but could also be a normal function in a module. It recieves some parameters, e.g. the data list and somehow formats it (for example it may add additional view parameters so the template can have less logic). It then returns the data set to the original view function which can either pass it along or process it further.
The second one is used for most other logic which I like to keep out of my direct view code for easier testing. My example of check_login does this: It is a function that is not directly tied to display output as its purpose is to check the users login credentials and decide to either return a user or report a login failure (by throwing an exception, return False or returning None). However, this functionality is not directly tied to a model either, so it cannot live inside an ORM class (well it could be a staticmethod for the User object). Instead it is just a function inside a module (remember, this is Python, you should use the simplest approach available, and functions are there for something)
To sum this up: Display logic in the view, all the other stuff in the model, since most logic is somehow tied to specific models. And if it is not, create a new module or package just for logic of this kind. This could be a separate module or even a package. For example, I often create a util module/package for helper functions, that are not directly tied for any view, model or else, for example a function to format dates that is called from the template but contains so much python could it would be ugly being defined inside a template.
Now we bring this logic to your task: Processing/Creation of letters. Since I don't know exactly what processing needs to be done, I can only give general recommendations based on my assumptions.
Let's say you have some data and want to bring it into a letter. So for example you have a list of articles and a costumer who bought these articles. In that case, you already have the data. The only thing that may need to be done before passing it to the template is reformatting it in such a way that the template can easily use it. For example it may be desired to order the purchased articles, for example by the amount, the price or the article number. This is something that is independent of the model, the order is now only display related (you could have specified the order already in your database query, but let's assume you didn't). In this case, this is an operation your view would do, so your template has the data ready formatted to be displayed.
Now let's say you want to get the data to create a specifc letter, for example a list of articles the user bough over time, together with the date when they were bought and other details. This would be the model's job, e.g. create a query, fetch the data and make sure it is has all the properties required for this specifc task.
Let's say in both cases you with to retrieve a price for the product and that price is determined by a base value and some percentages based on other properties: This would make sense as a model method, as it operates on a single product or order instance. You would then pass the model to the template and call the price method inside it. But you might as well reformat it in such a way, that the call is made already in the view and the template only gets tuples or dictionaries. This would make it easier to pass the same data out as an API (see below) but it might not necessarily be the easiest/best way.
A good rule for this decision is to ask yourself If I were to provide a JSON API additionally to my standard view, how would I need to modify my code to be as DRY as possible?. If theoretical is not enough at the start, build some APIs for the templates and see where you need to change things to the API makes sense next to the views themselves. You may never use this API and so it does not need to be perfect, but it can help you figure out how to structure your code. However, as you saw above, this doesn't necessarily mean that you should do preprocessing of the data in such a way that you only return things that can be turned into JSON, instead you might want to make some JSON specifc formatting for the API view.
So I went on a little longer than I intended, but I wanted to provide some examples to you because that is what I missed when I started and found out those things via trial and error.