We Keep Coding

How can I get the most recent records of a time series database?

I have a time series database with just three columns. The first column is for the id, the second is for the time stamps, and the third column is for the output of the iOT.
I am running the griddb python client on my Ubuntu machine. I need to get the most recent records from the database. The most recent records should be the ones just added on the day the query is run.
Whether it’s an SQL query or a function, I just need a way to make it work.
Here is how I stored the timeseries database in my GridDB container and an overview of the database.
##...
factory = griddb.StoreFactory.get_instance()
## Initializing the container
try:
gridstore = factory.get_store(host="127.0.0.1",
port=59111,
cluster_name="cluster_one",
username="admin",
password="")
conInfo = griddb.ContainerInfo("iOT_Project",
[["id", griddb.Type.INTEGER],
["time_stamp", griddb.Type.TIMESTAMP],
["output", griddb.Type.FLOAT]],
griddb.ContainerType.COLLECTION, True)
container = gridstore.put_container(conInfo)
## Reading the database with pandas
data = pd.read_csv("database.csv")
## Adding the database to the container
for i in range(len(data)):
ret = container.put(data.iloc[i, :])
print("Data added successfully")
except Exception as e:
print("Error: ", e)
"""
iot_database
id | time_stamp | output
------+--------------------------+---------------
232 | 01/01/2019 11:20 | 85309.52
233 | 02/01/2019 12:52 | 6398.54
234 | 03/01/2019 23:45 | 97583.63
235 | 04/01/2019 22:21 | 86533.11
236 | 05/01/2019 13:42 | 12421.35
"""

Output format of subprocess python

I'm trying to keep the format generated by a command to see all wordpress plugin list.
my script is :
import subprocess
r = open("/tmp/resultplugin", "w")
f = open("path/to/my/list", "r")
for x in f:
x = x.rstrip()
bashCommand = "wp plugin list --allow-root"
process = subprocess.Popen(bashCommand, cwd=x, shell=True, stdout=r)
output, error = process.communicate()
The output of the command when launched directly on the bash shell :
| name | status | update | version |
+---------------------------------+----------+-----------+------------+
| plugin name | inactive | none | 5.2.2 |
| plugin name | active | none | 10.4.0 |
| plugin name | inactive | none | 5.65 |
| plugin | inactive | none | 9.4.8 |
The output when i redirect the output in a file :
name status update version
plugin name inactive none 5.2.2
plugin name active none 10.4.0
plugin name inactive none 5.65
plugin name inactive none 9.4.8
I have hard time to find how to keep the same format, or at least have a little bit more visibility as it's very hard to read in the file
Can someone explain me how i can format the output to the file correctly ?
Thank you

How to handle environment prerequisites in BDD?

I am working on an automation test project (using Pytest BDD) and I constantly hit the problem of how to handle environment prerequisites using BDD and Gherkin. For example, almost all of the scenarios require new entities created (users/admins/sites/organizations/etc) only in order to have something to work with.
I think I shouldn't write all the prerequisite actions in the 'Given' section (it seems anti-BDD), but I don't want to get lost about what scenario sets up what and how.
For example, I would never want to create this:
Scenario: A user can buy a ticket from a webshop.
Given an item available in the webshop
And a user is created
And the user has at least one payment option set up
And the user is logged in
When the user buys the item in the webshop
Then the user owns the item
How people generally write down these actions and entities in a way that is readable and maintainable in the future?

One Way - Using #BeforeClass (One time setup)
#RunWith(Cucumber.class)
#CucumberOptions(features = "classpath:features/checkoutmodule/registereduser/",
glue = {"com.ann.automation.test.steps" },
tags = { "#SignIn" },
plugin = { "pretty","json:target/cucumber.json",
"junit:target/cucumber-reports/Cucumber.xml", "html:target/cucumber-reports",
"com.cucumber.listener.ExtentCucumberFormatter"},
strict = false,
dryRun = false,
monochrome = true)
public class RunCuke {
// ----------------------------- Extent Report Configuration -----------------------------
#BeforeClass
public static void setup() {
// below is dummy code just to showcase
File newFile = new File(Constants.EXTENT_REPORT_PATH);
ExtentCucumberFormatter.initiateExtentCucumberFormatter(newFile,true);
ExtentCucumberFormatter.loadConfig(new File(Constants.EXTENT_CONFIG_FILE_PATH));
ExtentCucumberFormatter.addSystemInfo("Browser Name", Constants.BROWSER);
ExtentCucumberFormatter.addSystemInfo("Browser version", Constants.BROWSER_VERSION);
ExtentCucumberFormatter.addSystemInfo("Selenium version", Constants.SELENIUM_VERSION);
}
}
Other Way - Using Background (Setup before every scenario)
Cucumber provides a mechanism for this, by providing a Background keyword
where you can specify
a step or series of steps which are common to all the tests in the feature file.
a step or series of steps should run before each scenario in the
feature. Typically these will be Given steps, but you can use any steps
that you need to.
Example : Here, before every scenario/outline execution, we want user to take to home page of the site and search for a product. So let's see the implementation.
Background:
Given User is on Brand Home Page "https://www.anntaylor.com/"
Given User searches for a styleId for <Site> and makes product selection on the basis of given color and size
| Style_ID | Product_Size | Product_Color |
| TestData1 | TestData1 | TestData1 |
| TestData2 | TestData2 | TestData2 |
#guest_search
Scenario Outline: Validation of UseCase Guest User Order Placement flow from Search
Then Clicking on Cart icon shall take user to Shopping Bag
When Proceeding to checkout as "GuestUser" with emailId <EmailID> shall take user to Shipping Page
And Entering FN as <FName> LN as <LName> Add as <AddL1> ZCode as <ZipCode> PNo as <PhoneNo> shall take user to payment page
And Submitting CCardNo as <CCNo>" Month as <CCMonth> Year as <CCYear> and CVV as <CVV> shall take user to Order Review Page
Then Verify Order gets placed successfully
Examples: Checkout User Information
| EmailID | FName | LName | AddL1 | ZipCode | PhoneNo | CCNo | CCMonth | CCYear | CVV |
| TestData2 | TestData2 | TestData2 | TestData2 | TestData2 | TestData2 | TestData2 | TestData2 | TestData2 | TestData2 |
Last Way - Using #Before (Setup before every scenario)
#Before
public void setUpScenario(Scenario scenario){
log.info("***** FEATURE FILE :-- " + Utility.featureFileName(scenario.getId().split(";")[0].replace("-"," ")) + " --: *****");
log.info("---------- Scenario Name :-- " + scenario.getName() + "----------");
log.info("---------- Scenario Execution Started at " + Utility.getCurrentTime() + "----------");
BasePage.message=scenario;
ExtentTestManager.startTest("Scenario No . " + (x = x + 1) + " : " + scenario.getName());
ExtentTestManager.getTest().log(Status.INFO, "Scenario No . "+ x + " Started : - " + scenario.getName());
// Utility.setupAUTTestRecorder();
// --------- Opening Browser() before every test case execution for the URL given in Feature File. ---------
BaseSteps.getInstance().getBrowserInstantiation();
}

Use Cucumber's "Background" keyword:
Background in Cucumber is used to define a step or series of steps which are common to all the tests in the feature file. It allows you to add some context to the scenarios for a feature where it is defined.
See also the official docs.

Can't seem to get pexpect to print data from command

I'm steadily working on how to ssh and parse data on a device by running a command. I have had a few questions along the way of this endeavor and much help with the questions I have asked. I'm now working with pexpect and I'm not seeing much in documentation with what I am doing. Basically I need to ssh in, as I said, and then run a command that will print out data, then get that data to print to my console.
Here is my code:
import pexpect
import pxssh
import getpass
child = pexpect.spawn('ssh www.example.com')
password = getpass.getpass('password: ')
child.sendline ('foo bar')
data = (child.read_nonblocking(size=1000, timeout=100))
print data
OUTPUT:
password:
foo bar
In the foo bar command the first line of the print out is foo bar so I am wondering if this is trying to print this data but only printing the first line. I add the read_nonblocking(size=1000, timeout=100) trying to set the size to be greater and a timeout to let the data print.
UPDATE with PXSSH
I have also tried to use the pxssh samples to do this and get only the list of commands that foo can run. I need to get the print out of foo bar which is the list of configs. My guess is that you can't have commands with spaces? Here is the code I have tried:
import pxssh
import getpass
try:
s = pxssh.pxssh()
s.force_password = True
hostname = raw_input('hostname: ')
username = raw_input('username: ')
password = getpass.getpass('password: ')
s.login (hostname, username, password)
s.sendline ('foo bar') # run a command
s.prompt() # match the prompt
print s.before # print everything before the prompt.
s.logout()
except pxssh.ExceptionPxssh, e:
print "pxssh failed on login."
print str(e)
Which gives me this back in console:
pxssh failed on login.
could not set shell prompt
:
Session idle time out is disabled
SSH> unset PROMPT_COMMAND
Error - Command [unset PROMPT_COMMAND] not found.
foo [ bar | bart | ran | up
| cmd | bee | hvac | monkey
| selective | list | help ]
check[v,nv,beep] [ list | help ]
delete [ all | bee | neewb | stuff
| up | cmd | fooconfig | root
| app | list | hvac | monkey
| selective | <filename> | confirmed | list | help ]
exit [ help ]
get [ vcf | nvcf | snmpcf | help ] [<filename>]
verbose [ help ]
help [ <command> | help ]
up arrow - brings up old command lines
down arrow - brings up newer command lines
right arrow - moves cursor to the right
left arrow - moves cursor to the left
insert - inserts a space at the cursor
delete - deletes character at the cursor
SSH> PS1='[PEXPECT]\$ '
Error - Command [PS1='[PEXPECT]\$ '] not found.
foo [ bar | bart | ran | up
| cmd | bee | hvac | monkey
| selective | list | help ]
check[v,nv,beep] [ list | help ]
delete [ all | bee | neewb | stuff
| up | cmd | fooconfig | root
| app | list | hvac | monkey
| selective | <filename> | confirmed | list | help ]
exit [ help ]
get [ vcf | nvcf | snmpcf | help ] [<filename>]
verbose [ help ]
help [ <command> | help ]
up arrow - brings up old command lines
down arrow - brings up newer command lines
right arrow - moves cursor to the right
left arrow - moves cursor to the left
insert - inserts a space at the cursor
delete - deletes character at the cursor
And as I mentioned I'm just trying to get the console to print out the foo bar command configs. This is the code I had working with python-exscript before finding out I needed to work in older Python 2.4.
CODE THAT I HAD WORKING IN EXSCRIPT THAT I NEED PEXPECT TO DO
account = read_login()
conn = SSH2()
conn.connect('example.com')
conn.login(account)
conn.execute('foo bar')
data = conn.response
conn.send('exit\r')
conn.close()
print data
Any help on how to get this code to work is greatly appreciated! Thanks!

Figured out the issue. I was missing a s.prompt()
try:
s = pxssh.pxssh(timeout=60, maxread=2000000)
s.force_password = True
hostname = raw_imput('hostname: ')
username = raw_input('password: ')
password = getpass.getpass('password: ')
s.PROMPT= 'SSH> '
s.login (hostname, username, password, auto_prompt_reset=False)
s.prompt()
s.sendline('foo bar')
s.prompt()
data = s.before
print data
s.logout()
except pxssh.ExceptionPxssh, e:
print "pxssh failed on login."
print str(e)

Python/Django/MySQL "Incorrect string value" error

I'm running a Django 1.4.2/Python 2.7.3/MySQL 5.5.28 site. One of the features of the site is that the admin can send an email to the server which calls a Python script via procmail that parses the email and tosses it into the DB. I maintain two versions of the site - a development and a production site. Both sites use different but identical vitualenvs (I even deleted them both and reinstalled all packages just to make sure).
I'm experiencing a weird issue. The exact same script succeeds on the dev server and fails on the production server. It fails with this error:
...django/db/backends/mysql/base.py:114: Warning: Incorrect string value: '\x92t kno...' for column 'message' at row 1
I'm well aware of the unicode issues Django has, and I know there are a ton of questions here on SO about this error, but I made sure to setup the database as UTF-8 from the beginning:
mysql> show variables like "character_set_database";
+------------------------+-------+
| Variable_name | Value |
+------------------------+-------+
| character_set_database | utf8 |
+------------------------+-------+
1 row in set (0.00 sec)
mysql> show variables like "collation_database";
+--------------------+-----------------+
| Variable_name | Value |
+--------------------+-----------------+
| collation_database | utf8_general_ci |
+--------------------+-----------------+
1 row in set (0.00 sec)
Additionally, I know that each column can have its own charset, but the message column is indeed UTF-8:
mysql> show full columns in listserv_post;
+------------+--------------+-----------------+------+-----+---------+----------------+---------------------------------+---------+
| Field | Type | Collation | Null | Key | Default | Extra | Privileges | Comment |
+------------+--------------+-----------------+------+-----+---------+----------------+---------------------------------+---------+
| id | int(11) | NULL | NO | PRI | NULL | auto_increment | select,insert,update,references | |
| thread_id | int(11) | NULL | NO | MUL | NULL | | select,insert,update,references | |
| timestamp | datetime | NULL | NO | | NULL | | select,insert,update,references | |
| from_name | varchar(100) | utf8_general_ci | NO | | NULL | | select,insert,update,references | |
| from_email | varchar(75) | utf8_general_ci | NO | | NULL | | select,insert,update,references | |
| message | longtext | utf8_general_ci | NO | | NULL | | select,insert,update,references | |
+------------+--------------+-----------------+------+-----+---------+----------------+---------------------------------+---------+
6 rows in set (0.00 sec)
Does anyone have any idea why I'm getting this error? Why is it happening under the production config but not the dev config?
Thanks!
[edit 1]
To be clear, the data are the same as well. I send a single email to the server, and procmail sends it off. This is what the .procmailrc looks like:
VERBOSE=off
:0
{
:0c
| <path>/dev/ein/scripts/process_new_mail.py dev > outputdev
:0
| <path>/prd/ein/scripts/process_new_mail.py prd > outputprd
}
There are 2 copies of process_new_mail.py, but that's just because it's version controlled so that I can maintain two separate environments. If I diff the two output files (which contain the message received), they're identical.
[edit 2]
I actually just discovered that both dev and prd configs are failing. The difference is that the dev config fails silently (maybe having to do with the DEBUG setting?). The problem is that there are some unicode characters in one of the messages, and Django is choking on them for some reason. I'm making progress....
I've tried editing the code to explicitly encode the message as ASCII and UTF-8, but it's still not working. I'm getting closer, though.

I fixed it! The problem was that I wasn't parsing the email correctly with respect to the charsets. My fixed email parsing code comes from this post and this post:
#get the charset of an email
#courtesy http://ginstrom.com/scribbles/2007/11/19/parsing-multilingual-email-with-python/
def get_charset(message, default='ascii'):
if message.get_content_charset():
return message.get_content_charset()
if message.get_charset():
return message.get_charset()
return default
#courtesy https://stackoverflow.com/questions/7166922/extracting-the-body-of-an-email-from-mbox-file-decoding-it-to-plain-text-regard
def get_body(message):
body = None
#Walk through the parts of the email to find the text body.
if message.is_multipart():
for part in message.walk():
#If part is multipart, walk through the subparts.
if part.is_multipart():
for subpart in part.walk():
if subpart.get_content_type() == 'text/plain':
#Get the subpart payload (i.e., the message body).
charset = get_charset(subpart, get_charset(message))
body = unicode(subpart.get_payload(decode=True), charset)
#Part isn't multipart so get the email body.
elif part.get_content_type() == 'text/plain':
charset = get_charset(subpart, get_charset(message))
body = unicode(part.get_payload(decode=True), charset)
#If this isn't a multi-part message then get the payload (i.e., the message body).
elif message.get_content_type() == 'text/plain':
charset = get_charset(subpart, get_charset(message))
body = unicode(message.get_payload(decode=True), charset)
return body
Thanks very much for the help!