Git: Version control intro#

“Revision control, also known as version control, source control manages changes to documents, programs, and other information stored as computer files – Wikipedia

Reproducibility?

  • Tracking and recreating every step of your work

  • In the software world: it’s called Version Control!

What do (good) version control tools give you?

  • Peace of mind (backups)

  • Freedom (exploratory branching)

  • Collaboration (synchronization)

Git is an enabling technology: Use version control for everything#

  • Write documents (never get paper_v5_john_jane_final_oct22_really_final.tex by email again!)

  • Write code

  • Backup your computer configuration

This tutorial is structured in the following way: we will begin with a brief overview of key concepts you need to understand in order for git to really make sense. We will then dive into hands-on work: after necessary configuration we will discuss 5 “stages of git” with scenarios of increasing sophistication and complexity, introducing the necessary commands for each stage:

  1. Local, single-user, linear workflow

  2. Single local user, branching

  3. Using remotes as a single user

  4. Remotes for collaborating in a small team

  5. distributed collaboration with large teams with GitHub/GitLab

Very high level picture: an overview of key concepts#

The commit: a snapshot of work at a point in time Every ball in this diagram represents a commit of all the files in a code repository, that we can go later in time, compare it with. We can also add labels/tags to this commits in case we want to develop new features.

image0

Credit: Gitflow Atlassian

And this is pretty much the essence of Git!

First things first: git must be configured before first use#

The minimal amount of configuration for git to work without pestering you is to tell it who you are:

Exercise#

The preceding ! marks that this code will execute in the terminal interpreter instead of in python

Lets tell Git who you are.

[1]:
!git config --global user.name "joaquin" # replace joaquin by your name
!git config --global user.email "user@gmail.com" # replace j by your email

And how you will edit text files (it will often ask you to edit messages and other information, and thus wants to know how you like to edit your files):

[2]:
# Put here your preferred editor. If this is not set, git will honor the $EDITOR environment variable
# On Windows: Notepad works, Notepad++, sublime or atom
# On mac/linux: vim, sublime or atom as a basic option
!git config --global core.editor nvim  # my lightweight neovim editor

# And while we're at it, we also turn on the use of color, which is very useful
!git config --global color.ui "auto"

Stage 0: Configure GIT#

Github is offers in its help pages instructions on how to configure the credentials helper for Mac OSX and Windows.

First we are going to create an SSH key that you can upload to Gitlab to recognize you. The key has a private part id_rsa that you should keep secret, and a public part id_rsa.pub that can upload it to Gitlab so gitlab knows it’s you.

Uncomment the following 2 lines, and make sure you run them only once, so you generate the key only one time.

[3]:
#! ssh-keygen -f ~/.ssh/id_rsa -t rsa -N ''
[4]:
#! less ~/.ssh/id_rsa.pub

Then paste the key into your Gitlab or github account

Stage 1: Local, single-user, linear workflow#

Simply type git to see a full list of all the ‘core’ commands. We’ll now go through most of these via small practical exercises:

[5]:
!git
usage: git [--version] [--help] [-C <path>] [-c <name>=<value>]
           [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
           [-p | --paginate | -P | --no-pager] [--no-replace-objects] [--bare]
           [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
           [--super-prefix=<path>] [--config-env=<name>=<envvar>]
           <command> [<args>]

These are common Git commands used in various situations:

start a working area (see also: git help tutorial)
   clone     Clone a repository into a new directory
   init      Create an empty Git repository or reinitialize an existing one

work on the current change (see also: git help everyday)
   add       Add file contents to the index
   mv        Move or rename a file, a directory, or a symlink
   restore   Restore working tree files
   rm        Remove files from the working tree and from the index

examine the history and state (see also: git help revisions)
   bisect    Use binary search to find the commit that introduced a bug
   diff      Show changes between commits, commit and working tree, etc
   grep      Print lines matching a pattern
   log       Show commit logs
   show      Show various types of objects
   status    Show the working tree status

grow, mark and tweak your common history
   branch    List, create, or delete branches
   commit    Record changes to the repository
   merge     Join two or more development histories together
   rebase    Reapply commits on top of another base tip
   reset     Reset current HEAD to the specified state
   switch    Switch branches
   tag       Create, list, delete or verify a tag object signed with GPG

collaborate (see also: git help workflows)
   fetch     Download objects and refs from another repository
   pull      Fetch from and integrate with another repository or a local branch
   push      Update remote refs along with associated objects

'git help -a' and 'git help -g' list available subcommands and some
concept guides. See 'git help <command>' or 'git help <concept>'
to read about a specific subcommand or concept.
See 'git help git' for an overview of the system.

We are going to create a test repo for git to play

git init: create an empty repository#

first we create a folder called playground_repo

[6]:
! mkdir playground_repo
[7]:
cd playground_repo
/home/runner/work/gdsfactory/gdsfactory/docs/notebooks/playground_repo

Let’s look at what git did:

[8]:
ls -la
total 8
drwxr-xr-x 2 runner docker 4096 Jun 29 01:29 ./
drwxr-xr-x 6 runner docker 4096 Jun 29 01:29 ../

The folder is empty now.

Lets create a new repo in the folder.

[9]:
! git init
hint: Using 'master' as the name for the initial branch. This default branch name
hint: is subject to change. To configure the initial branch name to use in all
hint: of your new repositories, which will suppress this warning, call:
hint: 
hint:      git config --global init.defaultBranch <name>
hint: 
hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
hint: 'development'. The just-created branch can be renamed via this command:
hint: 
hint:      git branch -m <name>
Initialized empty Git repository in /home/runner/work/gdsfactory/gdsfactory/docs/notebooks/playground_repo/.git/
[10]:
ls -la
total 12
drwxr-xr-x 3 runner docker 4096 Jun 29 01:29 ./
drwxr-xr-x 6 runner docker 4096 Jun 29 01:29 ../
drwxr-xr-x 7 runner docker 4096 Jun 29 01:29 .git/

Now you can see that there is a hidden folder .git (notice the dot that marks it as a hidden folder), which is the GIT repo

[11]:
ls -l .git
total 32
-rw-r--r-- 1 runner docker   23 Jun 29 01:29 HEAD
drwxr-xr-x 2 runner docker 4096 Jun 29 01:29 branches/
-rw-r--r-- 1 runner docker   92 Jun 29 01:29 config
-rwxr-xr-x 1 runner docker   73 Jun 29 01:29 description*
drwxr-xr-x 2 runner docker 4096 Jun 29 01:29 hooks/
drwxr-xr-x 2 runner docker 4096 Jun 29 01:29 info/
drwxr-xr-x 4 runner docker 4096 Jun 29 01:29 objects/
drwxr-xr-x 4 runner docker 4096 Jun 29 01:29 refs/

Now let’s edit our first file in the test directory with a text editor… I’m doing it programmatically here for automation purposes, but you’d normally be editing by hand

[12]:
!echo "My first bit of text in the repo" > README.md

ls lists the contents of the current working directory

[13]:
ls
README.md

git add: tell git about this new file#

[14]:
!git add README.md

We can now ask git about what happened with status:

[15]:
!git status
On branch master

No commits yet

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)
        new file:   README.md

git commit: permanently record our changes in git’s database#

For now, we are always going to call git commit either with the -a option or with specific filenames (git commit file1 file2...). This delays the discussion of an aspect of git called the index (often referred to also as the ‘staging area’) that we will cover later. Most everyday work in regular scientific practice doesn’t require understanding the extra moving parts that the index involves, so on a first round we’ll bypass it. Later on we will discuss how to use it to achieve more fine-grained control of what and how git records our actions.

[16]:
!git commit -a -m "First commit"
[master (root-commit) 0216568] First commit
 1 file changed, 1 insertion(+)
 create mode 100644 README.md

In the commit above, we used the -m flag to specify a message at the command line. If we don’t do that, git will open the editor we specified in our configuration above and require that we enter a message. By default, git refuses to record changes that don’t have a message to go along with them (though you can obviously ‘cheat’ by using an empty or meaningless string: git only tries to facilitate best practices, it’s not your nanny).

git log: what has been committed so far#

[17]:
!git log
commit 0216568d05be48876ef4d84507b53cc719a4c410 (HEAD -> master)
Author: joaquin <user@gmail.com>
Date:   Wed Jun 29 01:29:49 2022 +0000

    First commit

git diff: what have I changed?#

Let’s do a little bit more work… Again, in practice you’ll be editing the files by hand, here we do it via shell commands for the sake of automation (and therefore the reproducibility of this tutorial!)

[18]:
!echo "And now we add a second line..." >> README.md

And now we can ask git what is different:

[19]:
!git diff
diff --git a/README.md b/README.md
index db447fe..466ed60 100644
--- a/README.md
+++ b/README.md
@@ -1 +1,2 @@
 My first bit of text in the repo
+And now we add a second line...

The cycle of git virtue: work, commit, work, commit, …#

[20]:
!git commit -a -m "added second line."
[master 893e06d] added second line.
 1 file changed, 1 insertion(+)

git log revisited#

First, let’s see what the log shows us now:

[21]:
!git log
commit 893e06d83edfc6e29dd26d162e3b681c82c1f8c4 (HEAD -> master)
Author: joaquin <user@gmail.com>
Date:   Wed Jun 29 01:29:49 2022 +0000

    added second line.

commit 0216568d05be48876ef4d84507b53cc719a4c410
Author: joaquin <user@gmail.com>
Date:   Wed Jun 29 01:29:49 2022 +0000

    First commit

Sometimes it’s handy to see a very summarized version of the log:

[22]:
!git log --oneline --topo-order --graph
* 893e06d (HEAD -> master) added second line.
* 0216568 First commit

Git supports aliases: new names given to command combinations. Let’s make this handy shortlog an alias, so we only have to type git slog and see this compact log:

[23]:
# We create our alias (this saves it in git's permanent configuration file):
!git config --global alias.slog "log --oneline --topo-order --graph"

# And now we can use it
!git slog
* 893e06d (HEAD -> master) added second line.
* 0216568 First commit

git mv and rm: moving and removing files#

While git add is used to add fills to the list git tracks, we must also tell it if we want their names to change or for it to stop tracking them. In familiar Unix fashion, the mv and rm git commands do precisely this:

[24]:
!git mv README.md README.markdown
!git status
On branch master
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        renamed:    README.md -> README.markdown

Note that these changes must be committed too, to become permanent! In git’s world, until something hasn’t been committed, it isn’t permanently recorded anywhere.

[25]:
!git commit -a -m "I like this new name better"
!echo "Let's look at the log again:"
!git slog
[master fd59913] I like this new name better
 1 file changed, 0 insertions(+), 0 deletions(-)
 rename README.md => README.markdown (100%)
Let's look at the log again:
* fd59913 (HEAD -> master) I like this new name better
* 893e06d added second line.
* 0216568 First commit

And git rm works in a similar fashion removing the file from your repo.

Local user, branching#

What is a branch? A branch is a label for the state of a GIT repositories. It makes it easy to develop features and go back and forth between the original master and the copy feature branch version of the files inside the GIT repo

image0

Credit: Gitflow Atlassian

There can be multiple branches alive at any point in time; the working directory is the state of a special pointer called HEAD. In this example there are two branches, master and develop:

Once new commits are made on a branch, HEAD and the branch label move with the new commits:

This allows the history of both branches to diverge:

But based on this graph structure, git can compute the necessary information to merge the divergent branches back and continue with a unified line of development:

Let’s now illustrate all of this with a concrete example.

[26]:
!git status
On branch master
nothing to commit, working tree clean
[27]:
!ls
README.markdown

We are now going to try two different routes of development: on the master branch we will add one file and on the emojis branch, which we will create, we will add a different one. We will then merge the emojis branch into master.

[28]:
!git branch emojis
!git checkout emojis
Switched to branch 'emojis'
[29]:
!echo "Some emojis :smile:, :horse:, :cat:" > emojis.md
!git add emojis.md
!git commit -a -m "Adding some emojis"
!git slog
[emojis a5d66b6] Adding some emojis
 1 file changed, 1 insertion(+)
 create mode 100644 emojis.md
* a5d66b6 (HEAD -> emojis) Adding some emojis
* fd59913 (master) I like this new name better
* 893e06d added second line.
* 0216568 First commit
[30]:
!git checkout master
!git slog
Switched to branch 'master'
* fd59913 (HEAD -> master) I like this new name better
* 893e06d added second line.
* 0216568 First commit
[31]:
!ls
README.markdown

As you can see there are no emjois file in master yet

[32]:
!echo "All the while, more work goes on in master..." >> README.markdown
!git commit -a -m "The mainline keeps moving"
!git slog
[master da16096] The mainline keeps moving
 1 file changed, 1 insertion(+)
* da16096 (HEAD -> master) The mainline keeps moving
* fd59913 I like this new name better
* 893e06d added second line.
* 0216568 First commit
[33]:
ls
README.markdown
[34]:
!git merge emojis -m 'merge emojis'
!git slog
Merge made by the 'ort' strategy.
 emojis.md | 1 +
 1 file changed, 1 insertion(+)
 create mode 100644 emojis.md
*   8be6f23 (HEAD -> master) merge emojis
|\
| * a5d66b6 (emojis) Adding some emojis
* | da16096 The mainline keeps moving
|/
* fd59913 I like this new name better
* 893e06d added second line.
* 0216568 First commit

Using remotes as a single user#

We are now going to introduce the concept of a remote repository: a pointer to another copy of the repository that lives on a different location. This can be simply a different path on the filesystem or a server on the internet.

For this discussion, we’ll be using remotes hosted on the GitHub.com service, but you can equally use other services like Gitlab

Let’s see if we have any remote repositories here:

[35]:
!git remote -v

Since the above cell didn’t produce any output after the git remote -v call, it means we have no remote repositories configured. We will now proceed to do so. Once logged into Github or Gitlab, go to the new repository page and make a repository called test. Do not check the box that says Initialize this repository with a README, since we already have an existing repository here. That option is useful when you’re starting first at Github and don’t have a repo made already on a local computer.

We can now follow the instructions from the next page:

An important aside: conflict management#

While git is very good at merging, if two different branches modify the same file in the same location, it simply can’t decide which change should prevail. At that point, human intervention is necessary to make the decision. Git will help you by marking the location in the file that has a problem, but it’s up to you to resolve the conflict. Let’s see how that works by intentionally creating a conflict.

We start by creating a branch and making a change to our experiment file:

[36]:
!git branch trouble
!git checkout trouble
!echo "This is going to be a problem..." >> README.markdown
!git commit -a -m "Adding a file for trouble"
Switched to branch 'trouble'
[trouble 321ef24] Adding a file for trouble
 1 file changed, 1 insertion(+)

And now we go back to the master branch, where we change the same file:

[37]:
!git checkout master
!echo "At the same time master keeps working on same line will cause a MERGE CONFLICT ..." >> README.markdown
!git commit -a -m "Keep working on the experiment"
Switched to branch 'master'
[master 90c88f7] Keep working on the experiment
 1 file changed, 1 insertion(+)

So now let’s see what happens if we try to merge the trouble branch into master:

[38]:
!git checkout master
Already on 'master'
[39]:
!git merge trouble
Auto-merging README.markdown
CONFLICT (content): Merge conflict in README.markdown
Automatic merge failed; fix conflicts and then commit the result.

Let’s see what git has put into our file:

At this point, we go into the file with a text editor, decide which changes to keep, and make a new commit that records our decision. I’ve now made the edits, in this case I decided that both pieces of text were useful, but integrated them with some changes:

Let’s then make our new commit:

[40]:
#!git commit -a -m "Completed merge of trouble, fixing conflicts along the way"
#!git slog

Note: While it’s a good idea to understand the basics of fixing merge conflicts by hand, in some cases you may find the use of an automated tool useful. Git supports multiple merge tools: a merge tool is a piece of software that conforms to a basic interface and knows how to merge two files into a new one. Since these are typically graphical tools, there are various to choose from for the different operating systems, and as long as they obey a basic command structure, git can work with any of them.

Git resources#

this tutorial is based on Francisco Perez GIT notebook and has some ideas from the other links:

Photonics resources#

You can git clone some repositories for photonics and install the repo

Also, you can also add your fork information, git add, git commit, git push your changes and then send a pull request (request to merge your changes)

For photonics layout you can take a look at gdsfactory

and copy paste these lines below changing YourGithubUserName with you GitHub username

!git clone https://github.com/gdsfactory/gdsfactory.git
!cd gdsfactory
!make install
!git remote add fork git@github.com:YourGithubUserName/gdsfactory.git
[ ]:

For photonics layout you can take a look at UBCpdk

and copy paste these lines below changing YourGithubUserName with you GitHub username

!git clone https://github.com/gdsfactory/ubc.git
!cd ubc
!make install
!git remote add fork git@github.com:YourGithubUserName/ubc.git
[ ]:

For solving modes can take a look at modes

and copy paste these lines below changing YourGithubUserName with you GitHub username

!git clone https://github.com/joamatab/modesolverpy.git
!cd modesolverpy
!make install
!git remote add fork git@github.com:YourGithubUserName/modesolverpy.git
[ ]: