We Keep Coding

What is equivalent of Perl DB_FILE module in Python?

I was asked by my supervisor to convert some Perl scripts into Python language. I'm baffled by few lines of code and I am also relatively inexperienced with Python as well. I'm an IT intern, so this was something of a challenge.
Here are the lines of code:
my %sybase;
my $S= tie %sybase, "DB_File", $prismfile, O_RDWR|O_CREAT, 0666, $DB_HASH or die "Cannot open: $!\n";
$DB_HASH->{'cachesize' } = $cache;
I'm not sure what is the equivalent of this statement in Python? DB_FILE is a Perl module. DB_HASH is a database type that allows arbitrary keys/values to be stored in data file, at least that's according to Perl documentation.
After that, the next lines of code also got me stumped on how to convert this to the equivalent in Python as well.
$scnt=0;
while(my $row=$msth->fetchrow_arrayref()) {
$scnt++;
$srow++;
#if ($scnt <= 600000) {
$S->put(join('#',#{$row}[0..5]),join('#',#{$row}[6..19]));
perf(500000,'sybase') ;#if $VERBOSE ;
# }
}
I'll probably use fetchall() in Python to store the entire result dataset in it, then work through it row by row. But I'm not sure how to implement join() correctly in Python, especially since these lines use range within the row index elements -- [0..5]. Also it seems to write the output to data file (look at put()). I'm not sure what perf() does, can anyone help me out here?
I'd appreciate any kind of help here. Thank you very much.

Parsing specific keywords in Select Statements and formatting

I have a sample select statement:
Select D.account_csn, D.account_key, D.industry_id, I.industry_group_nm, I.industry_segment_nm From ecs.DARN_INDUSTRY I JOIN ecs.DARN_ACCOUNT D
ON I.SRC_ID=D.INDUSTRY_ID
WHERE D.ACCOUNT_CSN='5070000240'
I would like to parse the select statements into separate files. The first file name is called ecs.DARN_INDUSTRY
and inside the file it should look like this:
industry_group_nm
industry_segment_nm
Similarly another file called ecs.DARN_ACCOUNT and the content looks like this:
account_csn
account_key
industry_id
How do I do this in Bash or Python??

I doubt you will find a truly simple answer (maybe someone can prove otherwise). However, you might find python-sqlparse useful.
Parsing general SQL statments will be complicated and it is difficult to guess exactly what you are trying to accomplish. However, I think you are trying to extract the tables and corresponding column references via sql parsing, in which case, look at this question which basically asks that very thing directly.

Here is a long working command through awk,
awk 'NR==1{gsub(/^.*\./,"",$5);gsub(/^.*\./,"",$6);gsub(/.$/,"",$5); printf $5"\n"$6"\n" > "DARN_INDUSTRY"; gsub(/^.*\./,"",$2);gsub(/^.*\./,"",$3);gsub(/^.*\./,"",$4);gsub(/.$/,"",$2);gsub(/.$/,"",$3);gsub(/.$/,"",$4); printf $2"\n"$3"\n"$4"\n" > "DARN_ACCOUNT"}' file
Explanation:
gsub(/^.*\./,"",$5) remove all the characters upto the first . symbol in colum number 5.
printf $5"\n"$6"\n" > "DARN_INDUSTRY" redirects the output of printf command to the file named DARN_INDUSTRY.
gsub(/.$/,"",$4) Removes the last character in column 4.

CSV credential python one liner

I have a csv file like this :
name,username
name2,username2
etc...
And I need to extract each column into lists so I can create a account (admin script).
I am hoping the result would look like this :
NAMES=( name name2 )
MAILS=( username username2 )
LENGHT=3 # number of lines in csv files actually
I would like to do it in python (because I use it elsewhere in my script and would like to convert my collegues to the dark side). Exept that I am not really a python user...
Something like this would do the trick (I assume) :
NAMES=( $(echo "$csv" | pythonFooMagic) )
MAILS=( $(echo "$csv" | python -c "import sys,csv; pythonFooMagic2") )
LENGHT=$(echo "$csv" | pythonFooMagic3)
I kind of found tutos to do it accross several lines but glued together it was ugly.
There must be some cool ways to do it. Else I will resign to use sed... Any ideas?
EDIT : ok bad idea, for future reference, see the comments

You could use a temporary file, like this:
tmpfile=$(mktemp)
# Python script produces the variables you want
pythonFooMagic < csv > $tmpfile
# Here you take the variables somehow. For example...
source $tempfile
rm $tmpfile

processing text from a non-flat file (to extract information as if it *were* a flat file)

I have a longitudinal data set generated by a computer simulation that can be represented by the following tables ('var' are variables):
time subject var1 var2 var3
t1 subjectA ...
t2 subjectB ...
and
subject name
subjectA nameA
subjectB nameB
However, the file generated writes a data file in a format similar to the following:
time t1
description
subjectA nameA
var1 var2 var3
subjectB nameB
var1 var2 var3
time t2
description
subjectA nameA
var1 var2 var3
subjectB nameB
var1 var2 var3
...(and so on)
I have been using a (python) script to process this output data into a flat text file so that I can import it into R, python, SQL, or awk/grep it to extract information - an example of the type of information desired from a single query (in SQL notation, after the data is converted to a table) is shown below:
SELECT var1, var2, var3 FROM datatable WHERE subject='subjectB'
I wonder if there is a more efficient solution as each of these data files can be ~100MB each (and I have hundreds of them) and creating the flat text file is time-consuming and takes up additional hard drive space with redundant information. Ideally, I would interact with the original data set directly to extract the information that I desire, without creating the extra flat text file... Is there an awk/perl solution for such tasks that is simpler? I'm quite proficient at text-processing in python but my skills in awk are rudimentary and I have no working knowledge of perl; I wonder if these or other domain-specific tools can provide a better solution.
Thanks!
Postscript:
Wow, thanks to all! I am sorry that I cannot choose everyone's answers
#FM: thanks. My Python script resembles your code without the filtering step. But your organization is clean.
#PP: I thought I was already proficient in grep but apparently not! This is very helpful... but I think grepping becomes difficult when mixing the 'time' into the output (which I failed to include as a possible extraction scenario in my example! That's my bad).
#ghostdog74: This is just fantastic... but modifying the line to get 'subjectA' was not straightforward... (though I'll be reading up more on awk in the meantime and hopefully I'll grok later).
#weismat: Well stated.
#S.Lott: This is extremely elegant and flexible - I was not asking for a python(ic) solution but this fits in cleanly with the parse, filter, and output framework suggested by PP, and is flexible enough to accommodate a number of different queries to extract different types of information from this hierarchical file.
Again, I am grateful to everyone - thanks so much.

This is what Python generators are all about.
def read_as_flat( someFile ):
line_iter= iter(someFile)
time_header= None
for line in line_iter:
words = line.split()
if words[0] == 'time':
time_header = [ words[1:] ] # the "time" line
description= line_iter.next()
time_header.append( description )
elif words[0] in subjectNameSet:
data = line_iter.next()
yield time_header + data
You can use this like a standard Python iterator
for time, description, var1, var2, var3 in read_as_flat( someFile ):
etc.

If all you want is var1, var2, var3 upon matching a particular subject then you could try the following command:
grep -A 1 'subjectB'
The -A 1 command line argument instructs grep to print out the matched line and one line after the matched line (and in this case the variables come on a line after the subject).
You might want to use the -E option to make grep search for a regular expression and anchor the subject search to the beginning-of-line (e.g. grep -A 1 -E '^subjectB').
Finally the output will now consist of the subject line and variable line you want. You may want to hide the subject line:
grep -A 1 'subjectB' |grep -v 'subjectB'
And you may wish to process the variable line:
grep -A 1 'subjectB' |grep -v 'subjectB' |perl -pe 's/ /,/g'

The best option would be to modify the computer simulation to produce rectangular output. Assuming you can't do that, here's one approach:
In order to be able to use the data in R, SQL, etc. you need to convert it from hierarchical to rectangular one way or another. If you already have a parser that can convert the entire file into a rectangular data set, you are most of the way there. The next step is to add additional flexibility to your parser, so that it can filter out unwanted data records. Instead of having a file converter, you'll have a data extraction utility.
The example below is in Perl, but you can do the same thing in Python. The general idea is to maintain a clean separation between (a) parsing, (b) filtering, and (c) output. That way, you have a flexible environment, making it easy to add different filtering or output methods, depending on your immediate data-crunching needs. You can also set up the filtering methods to accept parameters (either from command line or a config file) for greater flexibility.
use strict;
use warnings;
read_file($ARGV[0], \&check_record);
sub read_file {
my ($file_name, $check_record) = #_;
open(my $file_handle, '<', $file_name) or die $!;
# A data structure to hold an entire record.
my $rec = {
time => '',
desc => '',
subj => '',
name => '',
vars => [],
};
# A code reference to get the next line and do some cleanup.
my $get_line = sub {
my $line = <$file_handle>;
return unless defined $line;
chomp $line;
$line =~ s/^\s+//;
return $line;
};
# Start parsing the data file.
while ( my $line = $get_line->() ){
if ($line =~ /^time (\w+)/){
$rec->{time} = $1;
$rec->{desc} = $get_line->();
}
else {
($rec->{subj}, $rec->{name}) = $line =~ /(\w+) +(\w+)/;
$rec->{vars} = [ split / +/, $get_line->() ];
# OK, we have a complete record. Now invoke our filtering
# code to decide whether to export record to rectangular format.
$check_record->($rec);
}
}
}
sub check_record {
my $rec = shift;
# Just an illustration. You'll want to parameterize this, most likely.
write_output($rec)
if $rec->{subj} eq 'subjectB'
and $rec->{time} eq 't1'
;
}
sub write_output {
my $rec = shift;
print join("\t",
$rec->{time}, $rec->{subj}, $rec->{name},
#{$rec->{vars}},
), "\n";
}

If you are lazy and have enough RAM, then I would work on a RAM disk instead of the file system as long as you need them immediately.
I do not think that Perl or awk will be faster than Python if you are just recoding your current algorithm into a different language.

awk '/time/{f=0}/subjectB/{f=1;next}f' file

Perl or Python script to remove user from group

I am putting together a Samba-based server as a Primary Domain Controller, and ran into a cute little problem that should have been solved many times over. But a number of searches did not yield a result. I need to be able to remove an existing user from an existing group with a command line script. It appears that the usermod easily allows me to add a user to a supplementary group with this command:
usermod -a -G supgroup1,supgroup2 username
Without the "-a" option, if the user is currently a member of a group which is not listed, the user will be removed from the group. Does anyone have a perl (or Python) script that allows the specification of a user and group for removal? Am I missing an obvious existing command, or well-known solution forthis? Thanks in advance!
Thanks to J.J. for the pointer to the Unix::Group module, which is part of Unix-ConfigFile. It looks like the command deluser would do what I want, but was not in any of my existing repositories. I went ahead and wrote the perl script using the Unix:Group Module. Here is the script for your sysadmining pleasure.
#!/usr/bin/perl
#
# Usage: removegroup.pl login group
# Purpose: Removes a user from a group while retaining current primary and
# supplementary groups.
# Notes: There is a Debian specific utility that can do this called deluser,
# but I did not want any cross-distribution dependencies
#
# Date: 25 September 2008
# Validate Arguments (correct number, format etc.)
if ( ($#ARGV < 1) || (2 < $#ARGV) ) {
print "\nUsage: removegroup.pl login group\n\n";
print "EXIT VALUES\n";
print " The removeuser.pl script exits with the following values:\n\n";
print " 0 success\n\n";
print " 1 Invalid number of arguments\n\n";
print " 2 Login or Group name supplied greater than 16 characters\n\n";
print " 3 Login and/or Group name contains invalid characters\n\n";
exit 1;
}
# Check for well formed group and login names
if ((16 < length($ARGV[0])) ||(16 < length($ARGV[1])))
{
print "Usage: removegroup.pl login group\n";
print "ERROR: Login and Group names must be less than 16 Characters\n";
exit 2;
}
if ( ( $ARGV[0] !~ m{^[a-z_]+[a-z0-9_-]*$}) || ( $ARGV[0] !~ m{^[a-z_]+[a-z0-9_-]*$} ) )
{
print "Usage: removegroup.pl login group\n";
print "ERROR: Login and/or Group name contains invalid characters\n";
exit 3;
}
# Set some variables for readability
$login=$ARGV[0];
$group=$ARGV[1];
# Requires the GroupFile interface from perl-Unix-Configfile
use Unix::GroupFile;
$grp = new Unix::GroupFile "/etc/group";
$grp->remove_user("$group", "$login");
$grp->commit();
undef $grp;
exit 0;

Web Link: http://www.ibm.com/developerworks/linux/library/l-roadmap4/
To add members to the group, use the gpasswd command with the -a switch and the user id you wish to add:
gpasswd -a userid mygroup
Remove users from a group with the same command, but a -d switch rather than -a:
gpasswd -d userid mygroup
"man gpasswd" for more info...
I looked for ages to find this. Sometimes it takes too much effort not to reinvent the wheel...

I found This for you. It should do what you need. As far as I can tell Perl does not have any built in functions for removing users from a group. It has several for seeing the group id of a user or process.

It looks like deluser --group [groupname] should do it.
If not, the groups command lists the groups that a user belongs to. It should be fairly straightforward to come up with some Perl to capture that list into an array (or map it into a hash), delete the unwanted group(s), and feed that back to usermod.

Here's a very simple little Perl script that should give you the list of groups you need:
my $user = 'user';
my $groupNoMore = 'somegroup';
my $groups = join ',', grep { $_ ne $groupNoMore } split /\s/, `groups $user`;
Getting and sanitizing the required arguments is left as an execrcise for the reader.