Related
I need to enhance my below script, which takes an input file that contains almost a million unique lines. Against each line, it has different values in 3 lookup files which I intend to add in my output as comma separated values.
The below script works fine, but it takes hours to finish the job. I am looking for a real fast solution which would also be less heavy on the system.
#!/bin/bash
while read -r ONT
do
{
ONSTATUS=$(grep "$ONT," lookupfile1.csv | cut -d" " -f2)
CID=$(grep "$ONT." lookupfile3.csv | head -1 | cut -d, -f2)
line1=$(grep "$ONT.C2.P1," lookupfile2.csv | head -1 | cut -d"," -f2,7 | sed 's/ //')
line2=$(grep "$ONT.C2.P2," lookupfile2.csv | head -1 | cut -d"," -f2,7 | sed 's/ //')
echo "$ONT,$ONSTATUS,$CID,$line1,$line2" >> BUwithPO.csv
} &
done < inputfile.csv
inputfile.csv contains the lines shown below:
343OL5:LT1.PN1.ONT1
343OL5:LT1.PN1.ONT10
225OL0:LT1.PN1.ONT34
225OL0:LT1.PN1.ONT39
343OL5:LT1.PN1.ONT100
225OL0:LT1.PN1.ONT57
lookupfile1.csv contains:
343OL5:LT1.PN1.ONT100, Down,Locked,No
225OL0:LT1.PN1.ONT57, Up,Unlocked,Yes
343OL5:LT1.PN1.ONT1, Down,Unlocked,No
225OL0:LT1.PN1.ONT34, Up,Unlocked,Yes
225OL0:LT1.PN1.ONT39, Up,Unlocked,Yes
lookupfile2.csv contains:
225OL0:LT1.PN1.ONT34.C2.P1, +123125302766,REG,DigitMap,Unlocked,_media_BNT,FD_BSFU.xml,
225OL0:LT1.PN1.ONT57.C2.P1, +123125334019,REG,DigitMap,Unlocked,_media_BNT,FD_BSFU.xml,
225OL0:LT1.PN1.ONT57.C2.P2, +123125334819,REG,DigitMap,Unlocked,_media_BNT,FD_BSFU.xml,
343OL5:LT1.PN1.ONT100.C2.P11, +123128994019,REG,DigitMap,Unlocked,_media_ANT,FD_BSFU.xml,
lookupfile3.csv contains:
343OL5:LT1.PON1.ONT100.SERV1,12-654-0330
343OL5:LT1.PON1.ONT100.C1.P1,12-654-0330
343OL5:LT7.PON8.ONT75.SERV1,12-664-1186
225OL0:LT1.PN1.ONT34.C1.P1.FLOW1,12-530-2766
225OL0:LT1.PN1.ONT57.C1.P1.FLOW1,12-533-4019
the output is:
225OL0:LT1.PN1.ONT57, Up,Unlocked,Yes,12-533-4019,+123125334019,FD_BSFU.xml,+123125334819,FD_BSFU.xml
225OL0:LT1.PN1.ONT34, Up,Unlocked,Yes,12-530-2766,+123125302766,FD_BSFU.xml,
343OL5:LT1.PN1.ONT1, Down,Unlocked,No,,,
343OL5:LT1.PN1.ONT100, Down,Locked,No,,,
343OL5:LT1.PN1.ONT10,,,,
225OL0:LT1.PN1.ONT39, Up,Unlocked,Yes,,,
As you'll see, the bottleneck will be executing grep within the loop multiple times. You can increase the efficiency by creating a look-up table with associative arrays.
If awk is available, please try the following:
[Update]
#!/bin/bash
awk '
FILENAME=="lookupfile1.csv" {
sub(",$", "", $1);
onstatus[$1] = $2
}
FILENAME=="lookupfile2.csv" {
split($2, a, ",")
if (sub("\\.C2\\.P1,$", "", $1)) line1[$1] = a[1]","a[6]
else if (sub("\\.C2\\.P2,$", "", $1)) line2[$1] = a[1]","a[6]
}
FILENAME=="lookupfile3.csv" {
split($0, a, ",")
if (match(a[1], ".+\\.ONT[0-9]+")) {
ont = substr(a[1], RSTART, RLENGTH)
cid[ont] = a[2]
}
}
FILENAME=="inputfile.csv" {
print $0","onstatus[$0]","cid[$0]","line1[$0]","line2[$0]
}
' lookupfile1.csv lookupfile2.csv lookupfile3.csv inputfile.csv > BUwithPO.csv
{EDIT]
If you need to specify absolute paths to the files, please try:
#!/bin/bash
awk '
FILENAME ~ /lookupfile1.csv$/ {
sub(",$", "", $1);
onstatus[$1] = $2
}
FILENAME ~ /lookupfile2.csv$/ {
split($2, a, ",")
if (sub("\\.C2\\.P1,$", "", $1)) line1[$1] = a[1]","a[6]
else if (sub("\\.C2\\.P2,$", "", $1)) line2[$1] = a[1]","a[6]
}
FILENAME ~ /lookupfile3.csv$/ {
split($0, a, ",")
if (match(a[1], ".+\\.ONT[0-9]+")) {
ont = substr(a[1], RSTART, RLENGTH)
cid[ont] = a[2]
}
}
FILENAME ~ /inputfile.csv$/ {
print $0","onstatus[$0]","cid[$0]","line1[$0]","line2[$0]
}
' /path/to/lookupfile1.csv /path/to/lookupfile2.csv /path/to/lookupfile3.csv /path/to/inputfile.csv > /path/to/BUwithPO.csv
Hope this helps.
If as you have indicated in the comments, you are unable to use the solution provided by #tshiono due to lacking gensub provided by GNU awk, you can replace gensub with two calls to sub with a temporary variable to accomplish trimming the needed suffix.
Example:
awk '
FILENAME=="lookupfile1.csv" {
sub(",$", "", $1);
onstatus[$1] = $2
}
FILENAME=="lookupfile2.csv" {
split($2, a, ",")
if (sub("\\.C2\\.P1,$", "", $1)) line1[$1] = a[1]","a[6]
else if (sub("\\.C2\\.P2,$", "", $1)) line2[$1] = a[1]","a[6]
}
FILENAME=="lookupfile3.csv" {
split($0, a, ",")
# ont = gensub("(\\.ONT[0-9]+).*", "\\1", 1, a[1])
sfx = a[1]
sub(/^.*[.]ONT[^.]*/, "", sfx)
sub(sfx, "", a[1])
# cid[ont] = a[2]
cid[a[1]] = a[2]
}
FILENAME=="inputfile.csv" {
print $0","onstatus[$0]","cid[$0]","line1[$0]","line2[$0]
}
' lookupfile1.csv lookupfile2.csv lookupfile3.csv inputfile.csv > BUwithPO.csv
I have commented out the use of gensub in the portion related to FILENAME=="lookupfile3.csv" and replaced the gensub expression with two calls to sub using sfx (suffix) as the temporary variable.
Give it a try and let me know if you are able to use that.
Perl solution
The following script is similar to the awk solution but written in Perl.
Save it as filter.pl and make it executable.
#!/usr/bin/env perl
use strict;
use warnings;
my %lookup1;
my %lookup2_1;
my %lookup2_2;
my %lookup3;
while( <> ) {
if ( $ARGV eq 'lookupfile1.csv' ) {
# 225OL0:LT1.PN1.ONT34, Up,Unlocked,Yes
# ^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^
if (/^([^,]+),\s*(.*)$/) {
$lookup1{$1} = $2;
}
} elsif ( $ARGV eq 'lookupfile2.csv' ) {
# 225OL0:LT1.PN1.ONT34.C2.P1, +123125302766,REG,DigitMap,Unlocked,_media_BNT,FD_BSFU.xml,
# ^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^ ^^^^^^^^^^^
if (/^(.+ONT\d+)\.C2\.P1,\s*([^,]+),(?:[^,]+,){4}([^,]+)/) {
$lookup2_1{$1} = "$2,$3";
} elsif (/^(.+ONT\d+)\.C2\.P2,\s*([^,]+),(?:[^,]+,){4}([^,]+)/) {
$lookup2_2{$1} = "$2,$3";
}
} elsif ( $ARGV eq 'lookupfile3.csv' ) {
# 225OL0:LT1.PN1.ONT34.C1.P1.FLOW1,12-530-2766
# ^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^
if (/^(.+ONT\d+)[^,]+,\s*(.*)$/) {
$lookup3{$1} = $2;
}
} else { # assume 'inputfile.csv'
no warnings 'uninitialized'; # because not all keys ($_) have values in the lookup tables
# 225OL0:LT1.PN1.ONT34
chomp;
print "$_,$lookup1{$_},$lookup3{$_},$lookup2_1{$_},$lookup2_2{$_}\n";
}
}
Execute it like so:
./filter.pl lookupfile{1,2,3}.csv inputfile.csv > BUwithPO.csv
It is important that the lookupfiles come first (as in the awk solutions, btw.) because
they build the four dictionaries (hashes in Perl parlance) %lookup1, %lookup2_1, etc.
and then the values from inputfile.csv are matched against those dictionaries.
here is my code :
type CatMixing struct {
Id bson.ObjectId `json:"id" bson:"_id,omitempty"`
CatMix []string `json:"comb"`
}
func main(){
session, err := mgo.Dial("127.0.0.1")
if err != nil {
panic(err)
}
defer session.Close()
session.SetMode(mgo.Monotonic, true)
c := session.DB("MixiIng").C("Combination")
var results []bson.M
err5 := c.Find(nil).Limit(10).All(&results)
if err5 == nil {
}
fmt.Println(results)
for _,catm := range results {
fmt.Println(catm)
for _,catm2 := range catm {
fmt.Println(reflect.TypeOf(catm2))
}
}
}
the problem is that it seems that comb is an array of interfaces :
map[_id:ObjectIdHex("590e5cb36aace327da180c89") comb:[60fps]]
bson.ObjectId
[]interface {}
but in mongo it shows as an array of string :
So my mapping is not working ...
If i try with :
var results []CatMixing
i have the _id but not comb , comb appears as empty
I don't understand why it's not an array of string and why my mapping is not working.
I've added data to mongodb with python :
from pymongo import MongoClient
client = MongoClient()
db = client['MixiIng']
collection = db['Combination']
combination = {}
result = [["60fps"]]
for r in result :
combination = {"comb":r}
collection.insert_one(combination)
So i don't understand why comb is not an array of string and how to get it...
thanks and regards
First you can change the results variable from the query using your []CatMixing. Because .All(result interface{}) needs an interace{} argument doesn't mean you can't pass your struct.
Note that interface{} in Go can hold any type including your struct.
try this code :
var results [] CatMixing
err := c.Find(bson.M{}).Limit(10).All(&results)
if err != nil {
fmt.Fatal(err)
}
fmt.Println(results)
So, i am trying to make a simple AOT virtual machine in golang, that reads a bytecode file as it's input. I am trying to very basically, write bytes to a file then read them with ioutil, however I am encountering a null dereference error.
This is my python code used for writing to the file:
btest = open("test.thief", "w")
bytes_to_write = bytearray([1, 44, 56, 55, 55, 0])
btest.write(bytes_to_write)
btest.close()
This is the code in my go file that I am using to read the bytes
package main
import (
"fmt"
"io/ioutil"
"os"
)
func main() {
//gets command line args
userArgs := os.Args[1:]
bytes, err := ioutil.ReadFile(userArgs[0]) // just pass the file name
if err != nil {
fmt.Print(err)
}
fmt.Println(bytes)
machine := NewEnv()
ReadBytes(machine, bytes)
}
And this is the error i am getting:
Joshuas-MacBook-Pro-3:thief Josh$ ./bin/Thief test.thief
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x2af3]
goroutine 1 [running]:
main.ReadBytes(0xc82000e390, 0xc82000c480, 0x6, 0x206)
/Users/Josh/thief/src/Thief/read.go:26 +0x353
main.main()
/Users/Josh/thief/src/Thief/Main.go:18 +0x1f2
I have also tried opening the file in binary mode with the python code, but it produces the same error. Basically the goal of this is that I want to be able to extract an array of bytes but also have a way to write theme.
Am i writing the bytes incorrectly?
Here is the hexdump:
012c 3837 3700
EDIT:
This is the rest of my go files
opfuncs.go
package main
//file for machine functions and operations
//prints string format of some bytes
func print(env Env, data []byte) {
fmt.Println(string(data))
}
var OPERS map[byte]func(Env, []byte)
//0 is reserved as the null terminator
func init(){
OPERS = map[byte]func(Env, []byte){
1:print,
}
}
env.go
package main
//file for the env struct
type Env struct {
items map[string]interface{}
}
func NewEnv() Env {
return Env{make(map[string]interface{})}
}
//general getter
func (self *Env) get(key string) interface{} {
val, has := self.items[key]
if has {
return val
} else {
return nil
}
}
//general setter
func (self *Env) set(key string, value interface{}) {
self.items[key] = value
}
func (self *Env) contains(key string) bool {
_, has := self.items[key]
return has
}
func (self *Env) declare(key string) {
self.items[key] = Null{}
}
func (self *Env) is_null(key string) bool {
_, ok := self.items[key].(Null)
return ok
}
//deletes an item from storage
func (self *Env) del(key string) {
delete(self.items, key)
}
read.go
package main
//file that contains the reading function for the VM
func ReadBytes(env Env, in []byte) {
bytes := make([]byte, 1)
fn := byte(0)
mode := 0
for i:=0;i<len(in);i++ {
switch mode {
case 0:
fn = byte(i)
mode = 1
case 1:
if i != len(in)-1 {
if in[i+1] == 0 {
bytes = append(bytes, in[i])
mode = 2
} else {
bytes = append(bytes, in[i])
}
} else {
bytes = append(bytes, in[i])
}
case 2:
OPERS[fn](env, bytes)
mode = 0
fn = byte(0)
bytes = make([]byte, 1)
}
}
}
If you use Python 3, the file must be opened in the binary mode:
btest = open("test.thief", "wb")
It looks like you're not writing to the file in your python code?
I get
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: write() argument must be str, not bytearray
After your third line which would explain why golang can't find the file contents.
Also you're not using bytes so your gocode won't compile. Do this instead.
package main
import (
"fmt"
"io/ioutil"
"os"
)
func main() {
//gets command line args
userArgs := os.Args[1:]
_, err := ioutil.ReadFile(userArgs[0]) // just pass the file name
if err != nil {
fmt.Print(err)
}
}
In Python, you can do this:
"File {file} had error {error}".format(file=myfile, error=err)
or this:
"File %(file)s had error %(error)s" % {"file": myfile, "error": err}
In Go, the simplest option is:
fmt.Sprintf("File %s had error %s", myfile, err)
which doesn't let you swap the order of the parameters in the format string, which you need to do for I18N. Go does have the template package, which would require something like:
package main
import (
"bytes"
"text/template"
"os"
)
func main() {
type Params struct {
File string
Error string
}
var msg bytes.Buffer
params := &Params{
File: "abc",
Error: "def",
}
tmpl, _ := template.New("errmsg").Parse("File {{.File}} has error {{.Error}}")
tmpl.Execute(&msg, params)
msg.WriteTo(os.Stdout)
}
which seems like a long way to go for an error message. Is there a more reasonable option that allows me to give string parameters independent of order?
With strings.Replacer
Using strings.Replacer, implementing a formatter of your desire is very easy and compact.
func main() {
file, err := "/data/test.txt", "file not found"
log("File {file} had error {error}", "{file}", file, "{error}", err)
}
func log(format string, args ...string) {
r := strings.NewReplacer(args...)
fmt.Println(r.Replace(format))
}
Output (try it on the Go Playground):
File /data/test.txt had error file not found
We can make it more pleasant to use by adding the brackets to the parameter names automatically in the log() function:
func main() {
file, err := "/data/test.txt", "file not found"
log2("File {file} had error {error}", "file", file, "error", err)
}
func log2(format string, args ...string) {
for i, v := range args {
if i%2 == 0 {
args[i] = "{" + v + "}"
}
}
r := strings.NewReplacer(args...)
fmt.Println(r.Replace(format))
}
Output (try it on the Go Playground):
File /data/test.txt had error file not found
Yes, you could say that this only accepts string parameter values. This is true. With a little more improvement, this won't be true:
func main() {
file, err := "/data/test.txt", 666
log3("File {file} had error {error}", "file", file, "error", err)
}
func log3(format string, args ...interface{}) {
args2 := make([]string, len(args))
for i, v := range args {
if i%2 == 0 {
args2[i] = fmt.Sprintf("{%v}", v)
} else {
args2[i] = fmt.Sprint(v)
}
}
r := strings.NewReplacer(args2...)
fmt.Println(r.Replace(format))
}
Output (try it on the Go Playground):
File /data/test.txt had error 666
A variant of this to accept params as a map[string]interface{} and return the result as a string:
type P map[string]interface{}
func main() {
file, err := "/data/test.txt", 666
s := log33("File {file} had error {error}", P{"file": file, "error": err})
fmt.Println(s)
}
func log33(format string, p P) string {
args, i := make([]string, len(p)*2), 0
for k, v := range p {
args[i] = "{" + k + "}"
args[i+1] = fmt.Sprint(v)
i += 2
}
return strings.NewReplacer(args...).Replace(format)
}
Try it on the Go Playground.
With text/template
Your template solution or proposal is also way too verbose. It can be written as compact as this (error checks omitted):
type P map[string]interface{}
func main() {
file, err := "/data/test.txt", 666
log4("File {{.file}} has error {{.error}}", P{"file": file, "error": err})
}
func log4(format string, p P) {
t := template.Must(template.New("").Parse(format))
t.Execute(os.Stdout, p)
}
Output (try it on the Go Playground):
File /data/test.txt has error 666
If you want to return the string (instead of printing it to the standard output), you may do it like this (try it on the Go Playground):
func log5(format string, p P) string {
b := &bytes.Buffer{}
template.Must(template.New("").Parse(format)).Execute(b, p)
return b.String()
}
Using explicit argument indices
This was already mentioned in another answer, but to complete it, know that the same explicit argument index may be used arbitrary number of times and thus resulting in the same parameter substituted in multiple times. Read more about this in this question: Replace all variables in Sprintf with same variable
I don't know of any easy way of naming the parameters, but you can easily change the order of the arguments, using explicit argument indexes:
From docs:
In Printf, Sprintf, and Fprintf, the default behavior is for each formatting verb to format successive arguments passed in the call. However, the notation [n] immediately before the verb indicates that the nth one-indexed argument is to be formatted instead. The same notation before a '*' for a width or precision selects the argument index holding the value. After processing a bracketed expression [n], subsequent verbs will use arguments n+1, n+2, etc. unless otherwise directed.
Then you can, ie:
fmt.Printf("File %[2]s had error %[1]s", err, myfile)
The parameter can also be a map, so the following function would work if you don't mind parsing every error format every time you use it:
package main
import (
"bytes"
"text/template"
"fmt"
)
func msg(fmt string, args map[string]interface{}) (str string) {
var msg bytes.Buffer
tmpl, err := template.New("errmsg").Parse(fmt)
if err != nil {
return fmt
}
tmpl.Execute(&msg, args)
return msg.String()
}
func main() {
fmt.Printf(msg("File {{.File}} has error {{.Error}}\n", map[string]interface{} {
"File": "abc",
"Error": "def",
}))
}
It's still a little wordier than I would have liked, but it's better than some other options, I suppose. You could turn map[string]interface{} into a local type and reduce it further to:
type P map[string]interface{}
fmt.Printf(msg("File {{.File}} has error {{.Error}}\n", P{
"File": "abc",
"Error": "def",
}))
Alas, there's no built-in function in Go for string interpolation with named parameters (yet). But you are not the only one suffering out there :) Some packages should exist, for example: https://github.com/imkira/go-interpol . Or, if feeling adventurous, you could write such a helper yourself, as the concept is actually quite simple.
Cheers,
Dennis
You can try the Go Formatter library that implements replacement fields surrounded by curly braces {} format strings similar to Python format.
Working code example Go Playground:
package main
import (
"fmt"
"gitlab.com/tymonx/go-formatter/formatter"
)
func main() {
formatted, err := formatter.Format("Named placeholders {file}:{line}:{function}():", formatter.Named{
"line": 3,
"function": "func1",
"file": "dir/file",
})
if err != nil {
panic(err)
}
fmt.Println(formatted)
}
Output:
Named placeholders dir/file:3:func1():
Instead of using template.New, where you have to provide a template name, you
can just instantiate a template pointer:
package main
import (
"strings"
"text/template"
)
func format(s string, v interface{}) string {
t, b := new(template.Template), new(strings.Builder)
template.Must(t.Parse(s)).Execute(b, v)
return b.String()
}
func main() {
params := struct{File, Error string}{"abc", "def"}
println(format("File {{.File}} has error {{.Error}}", params))
}
Use os.Expand to replace fields in a format string. Expand replaces ${var} or $var in the string using a func(string) string mapping function.
Here are a couple of ways to wrap os.Expand in convenient to use functions:
func expandMap(s string, m map[string]string) string {
return os.Expand(s, func(k string) string { return m[k] })
}
func expandArgs(s string, kvs ...string) string {
return os.Expand(s, func(k string) string {
for i := 1; i < len(kvs); i++ {
if kvs[i-1] == k {
return kvs[i]
}
}
return ""
})
}
Example use:
s = expandMap("File ${file} had error ${error}",
map[string]string{"file": "myfile.txt", "error": "Not found"})
s = expandArgs("File ${file} had error ${error}",
"file", "myfile.txt", "error", "Not found"))
Run the code on the playground.
You can get quite close to that sweet python formatting experience:
message := FormatString("File {file} had error {error}", Items{"file"=myfile, "error"=err})
Declare the following somewhere in your code:
type Items map[string]interface{}
func FormatString(template string, items Items) string {
for key, value := range items {
template = strings.ReplaceAll(template, fmt.Sprintf("{%v}", key), fmt.Sprintf("%v", value))
}
return template
}
💡 note that my implementation is very naive and inefficient for high-performance needs
sudo make me a package
Seeing the development experience potential with having a simple signature like this, I've got tempted and uploaded a go package called format.
package main
import (
"fmt"
"github.com/jossef/format"
)
func main() {
formattedString := format.String(`hello "{name}". is lizard? {isLizard}`, format.Items{"name": "Mr Dude", "isLizard": false})
fmt.Println(formattedString)
}
https://repl.it/#jossef/format
text/template is interesting. I Provide some example below
Usage
func TestFString(t *testing.T) {
// Example 1
fs := &FString{}
fs.MustCompile(`Name: {{.Name}} Msg: {{.Msg}}`, nil)
fs.MustRender(map[string]interface{}{
"Name": "Carson",
"Msg": 123,
})
assert.Equal(t, "Name: Carson Msg: 123", fs.Data)
fs.Clear()
// Example 2 (with FuncMap)
funcMap := template.FuncMap{
"largest": func(slice []float32) float32 {
if len(slice) == 0 {
panic(errors.New("empty slice"))
}
max := slice[0]
for _, val := range slice[1:] {
if val > max {
max = val
}
}
return max
},
"sayHello": func() string {
return "Hello"
},
}
fs.MustCompile("{{- if gt .Age 80 -}} Old {{else}} Young {{- end -}}"+ // "-" is for remove empty space
"{{ sayHello }} {{largest .Numbers}}", // Use the function which you created.
funcMap)
fs.MustRender(Context{
"Age": 90,
"Numbers": []float32{3, 9, 13.9, 2.1, 7},
})
assert.Equal(t, "Old Hello 13.9", fs.Data)
}
Lib
package utils
import (
"text/template"
)
type Context map[string]interface{}
type FString struct {
Data string
template *template.Template
}
func (fs *FString) MustCompile(expr string, funcMap template.FuncMap) {
fs.template = template.Must(template.New("f-string").
Funcs(funcMap).
Parse(expr))
}
func (fs *FString) Write(b []byte) (n int, err error) {
fs.Data += string(b)
return len(b), nil
}
func (fs *FString) Render(context map[string]interface{}) error {
if err := fs.template.Execute(fs, context); err != nil {
return err
}
return nil
}
func (fs *FString) MustRender(context Context) {
if err := fs.Render(context); err != nil {
panic(err)
}
}
func (fs *FString) Clear() string {
// return the data and clear it
out := fs.Data
fs.Data = ""
return out
}
important document
https://golang.org/pkg/text/template/#hdr-Actions
Here is a function I wrote which replaces fields with strings in a map, similar to what you can do with Python. It takes a string which should have fields that look like ${field} and replaces them with any such keys in the given map like map['field']='value':
func replaceMap(s string,m *map[string]string) string {
r := regexp.MustCompile("\\${[^}]*}")
for x,i := range *m {
s = strings.Replace(s,"${"+x+"}",i,-1)
}
// Remove missing parameters
s = r.ReplaceAllString(s,"")
return s
}
Playground example:
https://go.dev/play/p/S5rF5KLooWq
1>How can I run the equivalent using pymongo?
a>
cfg = rs.conf()
b>
db.printSlaveReplicationInfo()
2>Using PyMongo, how can I get the details of other replica sets CLI output in the created clusters .
(Note:I have already successfully created cluster.Just I am writing a python script in primary to check the outputs of rs.conf() and db.printSlaveReplicationInfo() in all the replica sets inside cluster and parse the output.)
Any help on this regard is greatly appreciable.
The replica set configuration is stored in the "local" database in a collection called "system.replset" so the equivalent of rs.conf() would be db.system.replset.findOne() when db is local or its equivalent find_one() in Python.
db.printSlaveReplicationInfo() is a big more involved, but you can get all of that information in the local database as well.
It may be easier to get it via admin database command replSetGetStatus which returns a document containing oplog information for each member of the replica set, along with other details. Python MongoDB driver pymongo provides a method to run commands, so you can run it against the admin DB and parse out the output for information about where each member of the replica set is relative to the primary.
I'm basically going to give you a hint rather than directly answer it, because the full answer is to simply code it. But you probably are unaware that you can do this simple thing in the shell:
> db.printSlaveReplicationInfo
function () {
var startOptimeDate = null;
function getReplLag(st) {
assert( startOptimeDate , "how could this be null (getReplLag startOptimeDate)" );
print("\tsyncedTo: " + st.toString() );
var ago = (startOptimeDate-st)/1000;
var hrs = Math.round(ago/36)/100;
print("\t" + Math.round(ago) + " secs (" + hrs + " hrs) behind the primary ");
};
function getMaster(members) {
var found;
members.forEach(function(row) {
if (row.self) {
found = row;
return false;
}
});
if (found) {
return found;
}
};
function g(x) {
assert( x , "how could this be null (printSlaveReplicationInfo gx)" )
print("source: " + x.host);
if ( x.syncedTo ){
var st = new Date( DB.tsToSeconds( x.syncedTo ) * 1000 );
getReplLag(st);
}
else {
print( "\tdoing initial sync" );
}
};
function r(x) {
assert( x , "how could this be null (printSlaveReplicationInfo rx)" );
if ( x.state == 1 || x.state == 7 ) { // ignore primaries (1) and arbiters (7)
return;
}
print("source: " + x.name);
if ( x.optime ) {
getReplLag(x.optimeDate);
}
else {
print( "\tno replication info, yet. State: " + x.stateStr );
}
};
var L = this.getSiblingDB("local");
if (L.system.replset.count() != 0) {
var status = this.adminCommand({'replSetGetStatus' : 1});
startOptimeDate = getMaster(status.members).optimeDate;
status.members.forEach(r);
}
else if( L.sources.count() != 0 ) {
startOptimeDate = new Date();
L.sources.find().forEach(g);
}
else {
print("local.sources is empty; is this db a --slave?");
return;
}
}
I love REPL's, and much like python's own famous REPL you can just get a dump of what the implemented function does.
Simples.