Mahfuzur Rahman Khan

How to check a value in list and finding indices of values in Python

2020-06-04T18:00:00+00:00

Before going to original section first learn about python methods Enumerate():

Using Enumerate we can easily print index with value in python. This is python built-in method. Enumerate() method adds a counter to an iterable and returns it in a form of enumerate object.

# Python3 code to demonstrate 
# print list values with index
# using enumerate() 

# initializing list 
test_list = [10, 20, 30, 40, 50, 60] 

# print from default 0 index
print("print from default 0 index")
[print(count, value) for count, value in enumerate(test_list)]


# print from starting 100 index
print("print from custom starting 100 index")
[print(count, value) for count, value in enumerate(test_list, 100)]

Now check a value in list and finding indices of value examples

# Python3 code to demonstrate 
# finding indices of values 
# using enumerate() 

# initializing list 
test_list = [1, 3, 4, 3, 6, 7] 

print (f"Original list : {test_list}") 

# checking value 3 in the list or not 
print(f"3 in the list : {3 in test_list}")

# checking value 8 in the list or not 
print(f"8 in the list : {8 in test_list}")

# using count() 
# to check 3 elements occurs no of times in the list
print(f"No of occurence of 3 element in the list is : {test_list.count(3)}")

# using enumerate() 
# to find indices for 3 
res_list = [i for i, value in enumerate(test_list) if value == 3] 
		
# printing resultant list 
print (f"New indices list : {res_list}")

Some useful link that I have followed:

Note: If you have any query or any upgradation of my writing or any mistakes please comment and suggest me. You are warmly welcomed always.

Linux Basic and Tricky Command

2020-05-28T18:00:00+00:00

#Linux

How to show the current directory in linux

$ pwd
/var/www/projects

View Directory Tree Structure In Linux

$ sudo apt-get install tree
$ tree
$ tree /etc/  # tree directory name
/var/www/projects

Additional links:

Note: If you have any query or any upgradation of my writing or any mistakes please comment and suggest me. You are warmly welcomed always.

How to upload attachment in AWS S3 Bucket from link in Python

2019-09-30T00:00:00+00:00

I have worked in a Chatbot product Lily Health that helps women through facebook chatting. I need to implement a new feature that can receive attachment from facebook and display it. When a facebook user sends any kinds of attachment, facebook first uploads the attachement in their fbcdn then returns the attachment url link in webhooks. Facebook provides the attachment in webhook post data as json format is given below:

{
   'entry':[
      {
         'messaging':[
            {
               'timestamp':1569844017208,
               'message':{
                  'mid':'RiEeqOXoTsxQUYdACamMx1FsMuxYuZpLd2Ti_QQepc4VjRjf1jL9L-ff3G8RwWOR14MTS94Ckmp12mh0pQXg2Q',
                  'attachments':[
                     {
                        'payload':{
                           'url':'https://scontent.xx.fbcdn.net/v/t1.15752-9/71384073_2440261556255816_5718908083162316800_n.png?_nc_cat=101&_nc_oc=AQlqBPPDfOFmpQTGj5-zAqI_LXbeGGk1WPztdaNjQeIisgqHUL5A8aFhRzJPgU37tmfpvFogpa763U-5am9A_4tL&_nc_ad=z-m&_nc_cid=0&_nc_zor=9&_nc_ht=scontent.xx&oh=c40e387ab65015190feb4bce4f03b7f1&oe=5E29EC37'
                        },
                        'type':'image'
                     },
                     {
                        'payload':{
                           'url':'https://scontent.xx.fbcdn.net/v/t1.15752-9/71818349_482421819282412_5182082100815200256_n.png?_nc_cat=111&_nc_oc=AQn7peXAPEqJ5fivE0W3jnwhfcrkeuC44XSVw6yg84w_ipClORs9IJwgd4Eme0MsFgQZb0_XNCT3H4AkJlajjOwm&_nc_ad=z-m&_nc_cid=0&_nc_zor=9&_nc_ht=scontent.xx&oh=c772cac3fb42df19406e5ab712441819&oe=5DF13B7E'
                        },
                        'type':'image'
                     }
                  ]
               },
               'sender':{
                  'id':'2369507619774719'
               },
               'recipient':{
                  'id':'2100837706646730'
               }
            }
         ],
         'id':'2100837706646730',
         'time':1569844017767
      }
   ],
   'object':'page'
}

Facebook Messenger Platform supports the following four attachment types, specified in the attachment.type property of the message:

I have planned to store the attachment in AWS S3 Bucket from the facebook attachments payload url and upload the attachment S3 bucket url in my database so that I can display it later. Here I have uploaded the attachment folder wise consider the attachment type.

So First I have to create a S3 bucket and you can follow this tutorial Create an Amazon S3 Bucket. After creating the bucket I need to public the bucket and you can follow my tutorial to make S3 Bucket Public. From S3 bucket we need the following attributes

AWS_ACCESS_KEY_ID
AWS_SECRET_KEY
AWS_REGION
AWS_BUCKET_NAME
AWS_S3_ENDPOINT

I have written a python function upload_file_to_aws_s3 that takes two parameter one is url and another is file type and this function return upload attachment S3 Bucket url.

import boto3
import requests
import os
from urllib.parse import urlparse

AWS_ACCESS_KEY_ID = "AWS_ACCESS_KEY_ID"
AWS_SECRET_KEY = "AWS_SECRET_KEY"
AWS_REGION = "AWS_REGION"
AWS_BUCKET_NAME = "AWS_BUCKET_NAME"
AWS_S3_ENDPOINT = "AWS_S3_ENDPOINT"

def upload_file_to_aws_s3(url='', file_type='image'):
    file_url = ''
    # get the connection of AWS S3 Bucket
    s3 = boto3.resource(
        's3',
        aws_access_key_id = AWS_ACCESS_KEY_ID,
        aws_secret_access_key = AWS_SECRET_KEY,
        region_name = AWS_REGION
    )
    
    response = requests.get(url)
    if response.status_code==200:
        raw_data = response.content
        url_parser = urlparse(url)
        file_name = os.path.basename(url_parser.path)
        key = file_type + "/" + file_name

        try:
            # Write the raw data as byte in new file_name in the server
            with open(file_name, 'wb') as new_file:
                new_file.write(raw_data)
    
            # Open the server file as read mode and upload in AWS S3 Bucket.
            data = open(file_name, 'rb')
            s3.Bucket(AWS_BUCKET_NAME).put_object(Key=key, Body=data)
            data.close()
            
            # Format the return URL of upload file in S3 Bucjet
            file_url = 'https://%s.%s/%s' % (AWS_BUCKET_NAME, AWS_S3_ENDPOINT, key)
        except Exception as e:
            print("Error in file upload %s." % (str(e)))
        
        finally:
            # Close and remove file from Server
            new_file.close()
            os.remove(file_name)
            print("Attachment Successfully save in S3 Bucket url %s " % (file_url))
    else:
        print("Cannot parse url")
    return file_url

If your AWS_BUCKET_NAME is mrk-lily-bucket, AWS_S3_ENDPOINT is s3-ap-southeast-1.amazonaws.com and key is image/71384073_2440261556255816_5718908083162316800_n.png then return url will be:

https://mrk-lily-bucket.s3-ap-southeast-1.amazonaws.com/image/71384073_2440261556255816_5718908083162316800_n.png.

After uploading different type of file in your S3 bucket you will get below folder wise upload structure in that bucket:

Some useful link that I have followed:

https://simpleisbetterthancomplex.com/tutorial/2017/08/01/how-to-setup-amazon-s3-in-a-django-project.html

Note: If you have any query or any upgradation of my writing or any mistakes please comment and suggest me. You are warmly welcomed always.

Git Basic and The mistake I have done in Git

2019-09-29T23:00:00+00:00

#Install Git

How to installation git in Debian and Ubuntu

Install git via ubuntu terminal
$ sudo apt-get install git

Git version check
$ git --version
git version 2.17.1

#Git 1

Create a new repository and clone into local

When you create your repository you will get this

# You can push your first commit from your local project using this command. 
# Go to your local project and execute following command to push your first commit.
$ echo "# git-practice" >> README.md
$ git init
$ git add README.md
$ git commit -m "first commit"
$ git remote add origin https://github.com/mahfuz110244/git-practice.git
$ git push -u origin master

You can also clone this repository to your local then manage your project. Suppose we want to start a new django project for git practice. First go to your desired location in local.

# Go to following directory where we want to clone git-practice repository
$ cd /var/www/
# Clone git-practice repository
$ git clone https://github.com/mahfuz110244/git-practice.git
Cloning into 'git-practice'...
warning: You appear to have cloned an empty repository.
# Check your clone repository
$ ls
$ betterstories  flask  git-practice  lily  react  react-js  revera_api  robi-winit
# Then go to your git-practice directory
$ cd git-practice/

#Git 2

How to push your first commit in GIT?

In the empty project I will add a django project mysite. Lets do it.

# Create virtual environment
$ virtualenv -p python3.6 venv
Running virtualenv with interpreter /usr/bin/python3.6
Using base prefix '/usr'
New python executable in /var/www/git-practice/venv/bin/python3.6
Also creating executable in /var/www/git-practice/venv/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
# Activate virtual environment
$ source venv/bin/activate
# Install django
$ (venv) pip install django
# make a django project mysite
$ (venv) django-admin startproject mysite
# Add a requirements file
$ (venv) pip freeze > requirements.txt
# Add a gitignore file
$ (venv) touch .gitignore

Let check and add the git commit file

# Check for commited file
$ git status
# Add all file for commit
$ git add .
# Check again for added commited file
$ git status

Now commit and push the added file

# Let put your first commit
$ git commit -m "[master] my first commit with a django project mysite"
# Let push your commit to master branch providing your github username and password
$ git push origin master

After sucessfully push you will find this

Let add a README file

# Create a README.md file
$ touch README.md
# Let commit and push this README.md file
$ git add .
$ git commit -m "[master] add a README file"
$ git push origin master
# Again you need to provide your github username and password
# We can save username and password checkout Git 6

#Git 3

How to create new branch and feature development branch in Git?

In professional development we need to create minimum three major branch must. Branches name should like this

master - Production Server
stage - QA/Staging Server
develop - Developer default local server

Lets create this major branches

$ git checkout -b new-branch-name copy-branch-name
$ git checkout -b stage master
$ git checkout -b develop stage

Now we are in develop branch. Lets run the django project.

$ cd mysite/
$ python manage.py runserver 8000

We want to develop a django app polls. This is one of feature of this project. So we should make a feature branch copy from develop branch.

New branch name should be in this sequence username/token/branch-name-by-hyphen. Here

username - Owner of the branch
token - Branch type
branch-name-by-hyphen - Branch name

Token

feat - A feature branch
hotfix - Hotfix changes for production issues
bugfix - A bugfix branch
chore - Cleaning up / organizing the code
wip - Branched out for some WIP stuff that has to be pushed You can checkout this link for the convention of branch name. Convention

Lets create poll app feature branch.

# Checkout to new poll-app-feature branch 
$ git checkout -b mahfuz110244/feat/poll-app develop
# Create a django polls app
$ python manage.py startapp polls

Rest of the Polls app part, you can follow DJango official tutorial from this link Polls

Now commit and push this polls app.

#Git 4

How to merge branch in Git?

We will merge mahfuz110244/feat/poll-app branch to develop.

https://git-scm.com/book/en/v2/Git-Branching-Basic-Branching-and-Merging
# Merge mahfuz110244/feat/poll-app branch to develop
$ git checkout develop
$ git merge mahfuz110244/feat/poll-app
$ git push origin develop

# Merge develop branch to stage
$ git checkout stage
$ git merge develop
$ git push origin stage

# Similarly we can merge stage branch to master

#Git 5

How to save username and password in Git?

Every time we need to give credential when we push any commit to remote. We can save our username and password using below git command.

$ git config --global credential.helper store

#Git 6

How to unstage any added file in Git?

In our simple poll app, we’ll create two models: Question and Choice. A Question has a question and a publication date. A Choice has two fields: the text of the choice and a vote tally. Each Choice is associated with a Question.

# initial migration for Django App
$ python manage.py migrate
# Check for file changes that are not stage
$ git status
$ git add polls/models.py
# Check for commited file
$ git status

# Now unstage the models.py file
$ git reset HEAD polls/models.py

#Git 7

How to change the last commit log message in Git?

Let’s say you just committed and you made a mistake in your commit log message. So we can change the commit log message using git amend command until push to the remote.

# Change most recent Git commit message
$ git commit --amend -m "an updated commit message"

Sometimes we forget to add any file in the commit. But after commit we can add the missing file using git amend command.

# Edit models.py and enums.py git add models.py git commit 
# Realize you forgot to add the changes from main.py git add main.py 
$ git commit --amend --no-edit

#Git 8

How to rollback to last commit push or any commit push in Git?

First you need to know the last push commit id or desired commit id HEAD No that you want to rollback. Using git reflog you can get your HEAD No.

Reflog

Reference logs, or “reflogs” are a mechanism Git uses to record updates applied to tips of branches and other commit references. Reflog allows you to go back to commits even though they are not referenced by any branch or tag. After rewriting history, the reflog contains information about the old state of branches and allows you to go back to that state if necessary. Every time your branch tip is updated for any reason (by switching branches, pulling in new changes, rewriting history or simply by adding new commits), a new entry will be added to the reflog.

$ git reflog

# show output
66a1920 HEAD@{0}: commit (amend): [mahfuz110244/feat/poll-app] update models
044c2fd HEAD@{1}: commit (amend): [mahfuz110244/feat/poll-app] add only one file, forget to another
0a4b7d3 HEAD@{2}: commit (amend): [mahfuz110244/feat/poll-app] add only one file, forget to another
fa36d58 HEAD@{3}: commit (amend): [mahfuz110244/feat/poll-app] add only one file, forget to another
282ad96 HEAD@{4}: commit: (amend): [mahfuz110244/feat/poll-app] add only one file, forget to another
fd0d72b HEAD@{5}: commit (amend): [mahfuz110244/feat/poll-app] add only one file, forget to another
e2dac0a HEAD@{6}: commit: (amend): [mahfuz110244/feat/poll-app] add only one file, forget to another
4ce2732 HEAD@{7}: commit (amend): [mahfuz110244/feat/poll-app] give proper log message
01a644e HEAD@{8}: commit: [mahfuz110244/feat/poll-app] give wrong log message
e3a6a13 HEAD@{9}: commit (amend): [mahfuz110244/feat/poll-app] add enums in model
afa4e27 HEAD@{10}: commit: [mahfuz110244/feat/poll-app] forget to add one file and give wrong message
abcd935 HEAD@{11}: commit: [mahfuz110244/feat/poll-app] add Question and Choice models in Polls App
b125266 (origin/stage, origin/develop, stage, develop) HEAD@{12}: checkout: moving from stage to mahfuz110244/feat/poll-app
b125266 (origin/stage, origin/develop, stage, develop) HEAD@{13}: merge develop: Fast-forward
acc7800 (origin/master, master) HEAD@{14}: checkout: moving from develop to stage
b125266 (origin/stage, origin/develop, stage, develop) HEAD@{15}: merge mahfuz110244/feat/poll-app: Fast-forward
acc7800 (origin/master, master) HEAD@{16}: checkout: moving from mahfuz110244/feat/poll-app to develop
b125266 (origin/stage, origin/develop, stage, develop) HEAD@{17}: commit: [mahfuz110244/feat/poll-app] add poll app

Now if you want to rollback your last push

$ git reset --hard HEAD@{1}
$ git push -f

If you want to rollback HEAD No 6 e2dac0a this commit just replace the HEAD No

$ git reset --hard HEAD@{6}
$ git push -f

#Git 9

How to switch branch without commit in Git?

Sometimes we need to switch between branches because of we don’t want to commit this or we have no time because of some urgent work of different branch. We can do that using git stash command.

Stash

The git stash command takes your uncommitted changes (both staged and unstaged), saves them away for later use, and then reverts them from your working copy.

Git Stash Pop

You can reapply previously stashed changes with git stash pop. Popping your stash removes the changes from your stash and reapplies them to your working copy.

$ git stash
$ git checkout desire_branch
$ git checkout stash_branch
$ git stash pop
$ git stash list

Git Stash Apply

Alternatively, you can reapply the changes to your working copy and keep them in your stash with git stash apply

$ git stash
$ git checkout desire_branch
$ git checkout stash_branch
$ git stash apply
$ git stash list

Suppose you have stash multiple times. You need to go to specific stash. Suppose you have two stash and you want to pop stash no 1.

$ git stash
$ git stash list
$ git stash pop stash@{1}

#Git 10

How to delete a Git branch locally and remotely?

Sometimes we make mistake in branch creating like we need to create a new branch that copy from master branch but mistakely we have created branch from develop. So we need to delete that branch from locally. If we push this branch in remote then we also need to delete branch from remote.

# Delete from remote
$ git push -d <remote_name> <branch_name>
# Delete from local
$ git branch -d <branch_name>
# Note that in most cases the remote name is origin. In such a case you'll have to use the command like so.
$ git push -d origin <branch_name>
#To delete the local branch use one of the following:
$ git branch -d branch_name
$ git branch -D branch_name

The -d option is an alias for –delete, which only deletes the branch if it has already been fully merged in its upstream branch. You could also use -D, which is an alias for –delete –force, which deletes the branch “irrespective of its merged status.”

#Git 11

Git Rebase

According to the git documentation, the rebase command will reapply commits on top of another base tip. This definition might be a little daunting. It’s easier to explain rebase as a procedure that adds the changes of the current branch to the tail of another branch.

The primary reason for Git rebasing is to maintain a linear project history. In rebase, you change your branch that’s being rebased so it looks like it was branched off a new base instead of the original. The process rewrites the commits, so you’ll end up with different commit IDs.

The golden rule of git rebase is to never use it on public branches.

$ git checkout feature-branch
# Commit some task and then rebase master for latest codebase.
$ git rebase master

# Checkout master branch and merge feature branch
git checkout master
git merge feature-branch

Additional links:

https://benmarshall.me/git-rebase/

#Git 12

Git Merge vs Rebase

Merge takes all the changes in one branch and merges them into another branch in one commit.
Rebase says I want the point at which I branched to move to a new starting point

So when do you use either one?

Merge Let’s say you have created a branch for the purpose of developing a single feature. When you want to bring those changes back to master, you probably want merge (you don’t care about maintaining all of the interim commits).

Rebase A second scenario would be if you started doing some development feature branch and then another developer made an unrelated change and merge that to base branch. Rebase is quite often a good idea to do before merging because you can resolve any conflict in your feature branch with base branch.

Git Rebasing Example In this example, we will first create a test case with master and feature branch. Then we will do a standard merge. Next, we will recreate the test case and perform rebase and merge.

Creating Master and Feature Branches

Here is the scenario we will create:

In the above example, we are taking the following path:

# Commit A: we add a.txt file in the ‘master’ branch
git checkout master
touch a.txt
git add -A
git commit -m "Commit A: added a.txt"

# Commit B: we add b.txt file in the ‘master’ branch
touch b.txt
git add -A
git commit -m "Commit B: added b.txt"

# Commit C: we add c.txt file in the ‘master’ branch
touch c.txt
git add -A
git commit -m "Commit C: added c.txt"
git status

# We go to the ‘feature’ branch
git checkout -b feature master 
# Commit E: we modify a.txt in ‘feature’ branch
echo aaa > a.txt
git add -A
git commit -m "Commit E: modified a.txt"

# Commit F: we modify b.txt in ‘feature’ branch
echo bbb > b.txt
git add -A
git commit -m "Commit F: modified b.txt"

Simple Merge

Let’s use the log command to check both branches.

# Results for ‘master’:
$ git checkout master
# Switched to branch 'master'
 
$ git log --oneline
2bbde47 Commit C: added c.txt
b430ab5 Commit B: added b.txt
6f30e95 Commit A: added a.txt
 
$ ls
a.txt    b.txt    c.txt

# Results for ‘feature’:
$ git checkout feature
# Switched to branch 'feature'
 
$ git log --oneline
0286690 Commit F: modified b.txt
7c5c85e Commit E: modified a.txt
b430ab5 Commit B: added b.txt
6f30e95 Commit A: added a.txt
 
$ ls
a.txt    b.txt

# Notice how the feature branch does not have Commit C

# Now let’s run merge ‘feature’ branch with ‘master’ branch. You will be asked to enter a comment. 
# In the comment, add “Commit G:” at the beginning to make it easier to track.

$ git checkout master
# Switched to branch 'master'
 
$ git merge feature
Merge made by the 'recursive' strategy.
a.txt | 1 +
b.txt | 1 +
2 files changed, 2 insertions(+)

# Results for ‘master’:
$ git checkout master
Already on 'master'
 
$ git log --oneline
 d086ff9 Commit G: Merge branch 'feature'
 0286690 Commit F: modified b.txt
 7c5c85e Commit E: modified a.txt
 2bbde47 Commit C: added c.txt
 b430ab5 Commit B: added b.txt
 6f30e95 Commit A: added a.txt
 
 $ ls
 a.txt b.txt c.txt
 
# Results for ‘feature’:
$ git checkout feature
# Switched to branch 'feature'
 
$ git log --oneline
0286690 Commit F: modified b.txt
7c5c85e Commit E: modified a.txt
b430ab5 Commit B: added b.txt
6f30e95 Commit A: added a.txt
 
$ ls
a.txt    b.txt

In the ‘master’ branch, you will notice there is a new commit G that has merged the changes from ‘feature’ branch. Basically, the following action has taken place:

In the Commit G, all the changes from feature branch have been brought into the master branch. But the feature branch itself has remained untouched due to the merge process. Notice the hash of each commit. After the merge, E (7c5c85e) and F (0286690) commit has the same hash on the feature and master branch.

Merging with Rebasing

Let’s repeat step 1 to create the master and feature branches again.

# Results for ‘master’:
$ git checkout master
# Switched to branch 'master'
 
$ git log --oneline
7f573d8 Commit C: added c.txt
795da3c Commit B: added b.txt
0f4ed5b Commit A: added a.txt
 
$ ls
a.txt    b.txt    c.txt

#Results for ‘feature’:
$ git checkout feature
# Switched to branch 'feature'
 
$ git log --oneline
8ed0c4e Commit F: modified b.txt
6e12b57 Commit E: modified a.txt
795da3c Commit B: added b.txt
0f4ed5b Commit A: added a.txt
 
$ ls
a.txt b.txt
# Let’s rebase from the ‘feature’ branch.

$ git checkout feature
# Switched to branch 'feature'
 
$ git rebase master
First, rewinding head to replay your work on top of it...
Applying: Commit E: modified a.txt
Applying: Commit F: modified b.txt
Then merge ‘feature’ into ‘master’.

$ git checkout master
# Switched to branch 'master'
 
$ git merge feature
Updating 7f573d8..9efa1a3
Fast-forward
 a.txt | 1 +
 b.txt | 1 +
 2 files changed, 2 insertions(+)
 
# Results for ‘master’ branch:
$ git log --oneline
9efa1a3 Commit F: modified b.txt
8710174 Commit E: modified a.txt
7f573d8 Commit C: added c.txt
795da3c Commit B: added b.txt
0f4ed5b Commit A: added a.txt
 
$ ls
a.txt    b.txt    c.txt

#Results for ‘feature’ branch:
$ git checkout feature
# Switched to branch 'feature'
 
$ git log --oneline
9efa1a3 Commit F: modified b.txt
8710174 Commit E: modified a.txt
7f573d8 Commit C: added c.txt
795da3c Commit B: added b.txt
0f4ed5b Commit A: added a.txt
 
$ ls
a.txt    b.txt    c.txt

Notice that after the rebase and merge both branches are the same. Also, the hashes for E and F have changed in both branches. Basically, in the rebase scenario, this is what happened:

That’s why there is no new commit. The E and F commits have been recalculated and latched to the end of the ‘master’ branch.

Rebasing is a useful tool when you want to clean up the history of your work. However, there is a danger which has given birth to the golden rule.

Additional links:

#Git 13

Git Cherry Pick

Cherry picking in Git means to choose a commit from one branch and apply it onto another. When you are working with a team of developers on a medium to large sized project, managing the changes between a number of git branches can become a complex task. Sometimes you don’t want to merge a whole branch into another, and only need to pick one or two specific commits. This process is called cherry picking.

When should we use Cherry Picking?

Let’s say you are working in an project where you are making changes in a branch called new-features. You have already made a few commits but want to move just one of them into the master branch. From new-features branch run git log –oneline to get a better log of your commits history. Note that the commit id is what we need to start the cherry picking.

# Checkout the branch where you want to cherry pick the specific commits. In this case master branch:
$ git checkout master

# Now we can cherry pick from new-features branch:
$ git cherry-pick d467740
# This will cherry pick the commit with hash d467740 and add it as a new commit on the master branch. 
# Note: it will have a new (and different) commit ID in the master branch.

# If you want to cherry pick more than one commit in one go, you can add their commit IDs separated by a space:
$ git cherry-pick d467740 de906d4

# If the cherry picking gets halted because of conflicts, resolve them and
$ git cherry-pick --continue

# If you want to bail of this step out altogether, just type:
$ git cherry-pick --abort

# In case you needed to cherry pick a merge instead of a commit, you can use:
$ git cherry-pick -m 1 <commit-id>

Caution!

Cherry picking is commonly discouraged in developer community. The main reason is because it creates a duplicate commit with the same changes and you lose the ability to track the history of the original commit. If you can merge, then you should use that instead of cherry picking. Use it with caution!

Additional links:

Note: If you have any query or any upgradation of my writing or any mistakes please comment and suggest me. You are warmly welcomed always.

Python Basic Tricks

2019-09-29T23:00:00+00:00

#Python 1

How to check a Empty Dictionary in Python

You can easily check a python dictionary is Empty or not using bool operator. Suppose we have a dictionary dct.

If bool(dct) is True then dict is not Empty
If bool(dct) is False then dict is Empty

Check Empty Dictionary
$ dct = {}
$ bool(dct)
False
$ not dct
True

Check Not Empty Dictionary
$ dct = { "n": 10 }
$ bool(dct)
True
$ not dct
False

#Python 2

How to check a Memory Address in Python3

First consider some objects in python that we want to know their memory address:

$ a = [10,20]
$ b = 10
$ bus1 = Bus([10,20]) # instance of Bus Class

id(object):

Return the “identity” of an object. This is an integer which is guaranteed to be unique and constant for this object during its lifetime. Two objects with non-overlapping lifetimes may have the same id() value.

$ id(-2)
10910304
$ id(-1)
10910336
$ id(0)
10910368
$ id(1)
10910400
$ id(2)
10910432
$ id(a)
139790571389000
$ id(a[0])
10910688
$ id(a[1])
10911008
$ id(b)
10910688
>>>> id(bus1)
139790562576704

hex(x):

Convert an integer number to a lowercase hexadecimal string prefixed with “0x”. If x is not a Python int object, it has to define an index() method that returns an integer. Using hex we can easily know the address of the object reference. Some examples:

$ hex(255)
'0xff'
$ hex(-42)
'-0x2a'
$ hex(id(a))
'0x7f238759c048'
>>>> hex(id(a[0]))
'0xa67be0'
$ hex(id(a[1]))
'0xa67d20'
$ hex(id(b))
'0xa67be0'
$ hex(id(bus1))
'0x7f2386d34940'

Using object.repr() we can easily know the object memory address with type of object.

$ object.__repr__(a)
'<list object at 0x7f238759c048>'
$ object.__repr__(b)
'<int object at 0xa67be0>'
$ object.__repr__(bus1)
'<__main__.Bus object at 0x7f2386d34940>'

#Python 3

How to find frequencies of all its unique elements in Python list or string?

Using python Collection module Counter we can easily find frequencies of all its unique elements.

from collections import Counter 
s1 ='aabbbcccdeff'
c1 = Counter(s1) 
print("c1 :", c1) 
  
# Counter used in List to find frequencies of all its unique elements of list 
L1 =[1, 2, 1, 1, 4, 4, 4, 5, 6, 6, 3, 3, 0, 0] 
t1 = Counter(L1) 
  
print("t1 :", t1)

Note: If you have any query or any upgradation of my writing or any mistakes please comment and suggest me. You are warmly welcomed always.

How to use normal Filter together with SearchFilter on Django Rest Framework?

2018-05-21T00:11:33+00:00

For any kind of list like doctor list, hospital list, pharmacy list etc. we need to search throughout the list regarding on different parameters. Django Rest Framework(DRF) provides us a great efficient search technique. By using we can easily search throughout list.

Suppose I have a model in Django like this:

class Doctor(models.Model):
    doctor_id = models.IntegerField(primary_key=True)
    name = models.CharField(max_length=250)
    institute = models.CharField(max_length=500, blank=True)
    speciality = models.CharField(max_length=500, blank=True)
    address = models.CharField(max_length=250, blank=True)
    country = models.CharField(max_length=50, blank=True)
    latitude = models.FloatField(max_length=50, blank=True, default=0)
    longitude = models.FloatField(max_length=50, blank=True, default=0)
    phone = models.CharField(max_length=250, blank=True)
    degree = models.CharField(max_length=250, blank=True)
    field = models.CharField(max_length=250, blank=True)

    class Meta:
        ordering = ["name"]

We can search or filter list data based on doctor name, institute, speciality, address, country. The problem I have faced that I wanna filter the doctor list based on country but search on doctor name, institute, speciality, address simultaneously. I have found a very good solution that I will share with you.

DjangoFilterBackend class provides us the country based filter while SearchFilter class provides us the name, institute, speciality, address based search. So we have just combined this two. It will be very helpful before start seeing the official documentation on Filtering. I think you read the documentation and understand the DjangoFilterBackend SearchFilter. So now lets start combine this two.

In the view class you declare DjangoFilterBackend SearchFilter in filter_backends. For DjangoFilterBackend you declare country field in filter_fields and for SearchFilter you declare doctor name, institute, speciality, address. Use the following code in your view.py:

from rest_framework import generics
from rest_framework import filters
from django_filters.rest_framework import DjangoFilterBackend
class DoctorAPIView(generics.ListAPIView):
    queryset = Doctor.objects.all()
    serializer_class = DoctorSerializer
    filter_backends = (filters.SearchFilter, DjangoFilterBackend)
    search_fields = ('name', 'speciality', 'institute', 'address')
    filter_fields = ('country',)

This will allow the client to filter the items in the list by making queries such as:

http://example.com/api/doctor?country=Bangladesh&search=Mr.Rahim
http://example.com/api/doctor?country=Bangladesh&search=Cardiologist
http://example.com/api/doctor?country=Bangladesh&search=Appolo
http://example.com/api/doctor?country=Bangladesh&search=Dhaka

Here,

country is the filter_fields. You can add more fields.
search parameter are search_fields what you given. You can add more fields.

Some useful link that I have followed:

https://stackoverflow.com/questions/40417970/how-to-use-normal-filter-together-with-searchfilter-on-django-rest-framework

Note: If you have any query or any upgradation of my writing or any mistakes please comment and suggest me. You are warmly welcomed always.

How to Make AWS S3 Bucket Public Readable

2018-05-21T00:11:33+00:00

Recently I have been working in Personal Health Management App where medical record can be saved as image. For my production server I have used AWS S3 Bucket. The problem I have faced I can’t see the image from my mobile app due to aws s3 bucket policy restriction. So I have need to set my AWS S3 bucket policy.

There is a lot of AWS Bucket Policy. You can generate your policy using AWS Policy Generator. AWS S3 allows us to set permission on file basis or per bucket. In my case I need to make my bucket publicly readable for everyone. To set this you need to follow the below instruction:

Go to your AWS Management Console and you will see your buckets(see the image)
Select the bucket you wanna make public readable. Click the top permission tab and select below Bucket Policy tab(see the image).
Paste following code in the Bucket Policy Modal and save it.

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "AllowPublicRead",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": [
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::<your-bucket-name>/*"
      ]
    }
  ]
}

Don’t forget to put your bucket name in your-bucket-name option:

After save the bucket policy you will see that your bucket public(see the image).

Some useful link that I have followed:

https://futurestud.io/tutorials/how-to-make-aws-s3-bucket-public-readable

Note: If you have any query or any upgradation of my writing or any mistakes please comment and suggest me. You are warmly welcomed always.

How to Backup and Restore a AWS RDS PostgreSQL Database

2018-05-14T00:11:33+00:00

In my previous tutorial Database Backup I have discussed Backup and Restore a PostgreSQL Database. In this tutorial I will discuss How to Backup and Restore a AWS RDS PostgreSQL Database.

Amazon Relational Database Service Amazon RDS makes it easy to set up, operate, and scale a relational database in the cloud. It provides cost-efficient and resizable capacity while automating time-consuming administration tasks such as hardware provisioning, database setup, patching and backups. It frees you to focus on your applications so you can give them the fast performance, high availability, security and compatibility they need. To backup AWS RDS postgresql database we follow the following steps:

Configure RDS Security Group

Change your database RDS instance security group to allow your machine to access it. Add your ip to the security group to access the instance via Postgres. Follow the steps:

Go to your RDS Dashboard, select Instances and open the instance you want to connect to.
Look for a line like this : Security Groups rds-launch-wizard rds-launch-wizard (sg-0ef8c1f5c707d7e05)( active )(see the upper image). You should be able to click on rds-launch-wizard (sg-3e9axxx) which leads you to the EC2 Dashboard in the Security Groups panel.
Select your security group and open the Inbound Tab in the lower panel(see the upper image).
Click on Edit and Add Rule. Don’t forget to set the Port Range to 5432 and select My IP in the source.
Click on Save.
You should be able to connect to your RDS Instance from your local IP.

We will use pg_dump tool to backup database. pg_dump dumps a database as a text file or to other formats. It will dump all the contents of a selected database into a single file. We need to run pg_dump in the command line on the computer where the database is stored. So if the database is stored on a remote server, you will need to SSH to that server. Here I will discuss two of of backup database:

Database Backup

We backup database as a plain-text file format containing SQL script using pg_dump. Run the following command in the remote server terminal:

sudo pg_dump -h <public_dns> -U <db_user> -f </path/to/your/file/name_of_dump_file.sql> <db_name>

Here,

-h to specify AWS RDS public dns Endpoint.
-U to specify which user will connect to the PostgreSQL database server.
db_user means database username
-f is used to specify the output format of file.
db_name means database name
name_of_dump_file means backup database file name
.sql means backup database file as plain-text file containing SQL script.

If your AWS RDS public_dns is test_db.cnxjhgdlehkxv.ap-southeast-1.rds.amazonaws.com, db_user is khan, db_name is test_db and want to save the database in Downloads folder then

sudo pg_dump -h test_db.cnxjhgdlehkxv.ap-southeast-1.rds.amazonaws.com -U khan -W -F t test_db > /home/khan/Downloads/bk_test_db_18_05_2018.sql

bk_test_db_18_05_2018.sql contains all the SQL queries that are required to restore your database. You will find your dump file in the path you provide. Database backup part is done, lets move on next restoring part.

Copy Backup Database to Local

To restore database we need to copy the remote server backup dump file locally. For this run the following command locally:

scp -i </path/to/your/file/test_db.pem> username@remote:/file/to/send /where/to/put

If your username is user and remote server is 171.16.220.227 then

scp -i /home/bs086/Downloads/test_db.pem user@171.16.220.227:/home/khan/Downloads/bk_test_db_18_05_2018.sql /home/khan/Downloads

Now go to your Downloads folder you will see this bk_test_db_18_05_2018.sql file.

Database Restore

We will use psql for restoring from a plain SQL script file created with pg_dum. Run the following command in the local terminal to restore your plain-text file containing SQL script:

sudo psql -U <db_user> -d <new_created_db_name> -f </path/to/your/file/dump file that you want to restore>

Here,

-U to specify which user will connect to the PostgreSQL database server.
db_user means database username
-d to specify which new created database you will restore your backup database.
-f is used to specify the output format of file.

If your db_user is khan, newly created db is new_test_db and backup file is bk_test_db_02_05_2018.sql then run following command

sudo psql -U khan -d new_test_db -f /home/khan/Downloads/bk_test_db_02_05_2018.sql

After entering the command you will see some tamil/telegu programming UI. Now go to your management tool for PostgreSQL like pgadmin, [datagrip](https://www.jetbrains.com/datagrip/ “”) view and you will see the restore database.

See my the previous tutorial How to Backup and Restore a PostgreSQL Database

Some useful link that I have followed:

https://www.axiomq.com/blog/backup-and-restore-a-postresql-database/
http://stackoverflow.com/questions/31881786/how-to-pg-dump-an-rds-postgres-database
http://www.thegeekstuff.com/2009/01/how-to-backup-and-restore-postgres-database-using-pg_dump-and-psql/

Note: If you have any query or any upgradation of my writing or any mistakes please comment and suggest me. You are warmly welcomed always.

How to kill a process on a specific port on linux

2018-05-02T00:11:33+00:00

Which problem I have faced several time in linux that is I want to kill a specific port process ID(PID). But its difficult to find the process to check in the console. Run following command in the terminal to see the all process run in linux.

ps -ef

If you run the command you will see like that(see the image)

If you want to kill a process identification number(PID) just run the following command in terminal

sudo kill -9 PID

Here,

sudo - command to ask admin privilege(user id and password).
kill - command to kill the process
-9 - forcefully
PID - process identification number that you want to kill

If you want to kill PID 30 then command will be:

sudo kill -9 30

Sometimes I fed up with searching my program PID. As you know the port number so you can easily find the port PID and kill it. If you want to kill a process running on port number 8000 then first you need to find the PID and then kill it. Run the following command to find port number PID:

sudo lsof -t -i:8000

Here,

lsof - list of files(Also used for to list related processes)
-t - show only process ID
-i - show only internet connections related process
:8000 - show only processes in this port number

So you can now easily kill your PID. One command to to kill a process on a specific port use the following command:

sudo kill -9 $(sudo lsof -t -i:8000)

Note: If you have any query or any upgradation of my writing or any mistakes please comment and suggest me. You are warmly welcomed always.

How to Backup and Restore a PostgreSQL Database

2018-05-02T00:11:33+00:00

Its very important to backup and restore any kind of database. Suppose your testing server or production server have an issue but you can’t find the issue locally. So how can you check the issue and solve it. For solve the issue you need to backup the server database and restore it locally. After restoring you can check the issue and solve it. This problem have raised as your local and server configuration are not same always.

Database Backup

First Way

We backup database as a plain-text file format containing SQL script. Run the following command in the remote server terminal:

sudo pg_dump -U <db_user> -f </path/to/your/file/name_of_dump_file.sql> <db_name>

Here,

-U to specify which user will connect to the PostgreSQL database server.
db_user means database username
-f is used to specify the output format of file.
db_name means database name
name_of_dump_file means backup database file name
.sql means backup database file as plain-text file containing SQL script.

If your db_user is khan, db_name is test_db and want to save the database in Downloads folder then

sudo pg_dump -U khan -W -F t test_db > /home/khan/Downloads/bk_test_db_02_05_2018.sql

bk_test_db_02_05_2018.sql contains all the SQL queries that are required to restore your database. You will find your dump file in the path you provide. Database backup part is done, lets move on next restoring part.

Second Way

Run the following command in the remote server terminal:

sudo pg_dump -U db_user -W -F t db_name > /path/to/your/file/name_of_dump_file.tar

Here,

-U to specify which user will connect to the PostgreSQL database server.
db_user means database username
-W will force pg_dump to prompt for a password before connecting to the server.
-F is used to specify the format of the output file, which can be one of the following:
- p - plain-text SQL script
- c - custom-format archive
- d - directory-format archive
- t - tar-format archive
db_name means database name
name_of_dump_file.tar means backup database file name

To see list of command for pg_dump use pg_dump -? in the terminal

If your db_user is khan, db_name is test_db and want to save the database in Downloads folder then

sudo pg_dump -U khan -W -F t test_db > /home/khan/Downloads/bk_test_db_02_05_2018.tar

bk_test_db_02_05_2018.tar contains all the SQL queries that are required to restore your database. You will find your dump file in the path you provide.

Backup Database Copy to Local

To restore database we need to copy the remote server backup dump file locally. For this run the following command in the local terminal:

scp username@remote:/file/to/send /where/to/put

If your username is user and remote server is 171.16.220.227 then

scp user@171.16.220.227:/home/khan/Downloads/bk_test_db_02_05_2018.sql /home/khan/Downloads

Now go to your Downloads folder you will see this bk_test_db_02_05_2018.sql file.

Database Restore

There are two ways to restore a PostgreSQL database:

psql for restoring from a plain SQL script file created with pg_dump,
pg_restore for restoring from a .tar file, directory, or custom format created with pg_dump.

First Way: Restore a database with psql

Run the following command in the local terminal to restore your plain-text file containing SQL script:

sudo psql -U <db_user> -d <new_created_db_name> -f </path/to/your/file/dump file that you want to restore>

Here,

-U to specify which user will connect to the PostgreSQL database server.
db_user means database username
-d to specify which new created database you will restore your backup database.
-f is used to specify the output format of file.

If your db_user is khan, newly created db is new_test_db and backup file is bk_test_db_02_05_2018.sql then run following command

sudo psql -U khan -d new_test_db -f /home/khan/Downloads/bk_test_db_02_05_2018.sql

After entering the command you will see some tamil/telegu programming UI. Now go to your pgadmin view and you will see the restore database.

Second Way: Restore a database with pg_restore

Here we use pg_restore to restore our database locally. If you choose custom, directory, or archive format when creating a backup file, then you will need to use pg_restore in order to restore your database. Using the following command locally:

sudo pg_restore -d <new_created_db_name> </path/to/your/file/dump file that you want to restore> -c -U <db_user>

Here,

new_created_db_name means new created database where you wanna store backup database

If you use pg_restore you have various options available, for example:

-c to drop database objects before recreating them,
-C to create a database before restoring into it,
-e exit if an error has encountered,
-F format to specify the format of the archive.

Use pg_restore -? to get the full list of available options.

Restore our bk_test_db_02_05_2018.tar database command will be

sudo pg_restore -d new_test_db /home/user/Downloads/bk_database_02_05_2018.tar -c -U user

Now go to your management tool for PostgreSQL like pgadmin, [datagrip](https://www.jetbrains.com/datagrip/ “”) view and you will see the restore database.

See the next tutorial How to Backup and Restore a AWS RDS PostgreSQL Database

Some useful link that I have followed:

https://www.axiomq.com/blog/backup-and-restore-a-postresql-database/

Note: If you have any query or any upgradation of my writing or any mistakes please comment and suggest me. You are warmly welcomed always.