Working with multiple repositories

Programming
1

I’m wondering how people have been organizing their work on Kaleidoscope and its various plugins. Supposing I want to keep up to date with Arduino-Boards and all of its submodules, but I want to fork a subset of those modules to work on them, how should I manage that? Should I fork Arduino-Boards and edit .gitmodules to point to my forked repositories? Should I fork the whole damned thing? Is there a PATH location for the Arduino build system that lets me just drop my versions of things in and have them take precedence over the Arduino-Boards submodules?

Managing this is a bit of a pain for those of us who don’t get to work directly with the central repository, and I’m hoping someone has worked out a good system for doing so.

3 Likes
2

you might be able to do something with .kaleidoscope-builder.conf it lets yout pass some extra args to arduino-builder https://github.com/algernon/Model01-sketch/blob/master/.kaleidoscope-builder.conf @algernon is using it to pull in libraries not in the arduino-boards repo.

1 Like
3

@merlin can you open me a ticket on Kaleidoscope to document my workflow? I’ve found reasonably simple ways to do what i believe you want.

3 Likes
4

Done. And don’t rush it on my account; I can muddle through for a while without a well-defined procedure.

5

You are…not the first to express frustration with submodules. They’re the worst solution. Except for all the others, which are even worse. :wink:

3 Likes
6

My workflow is pretty simple: I cloned Arduino-Boards into a directory within my sketch, and every plugin I fork, gets a new remote, and I check out the branch I want. To keep up with upstream, I loop over the submodules, and git fetch --all each of them, and manually rebase/merge whenever I feel like it.

This works wonderfully as long as you only need the stuff locally (as is my case). If you want to publish an Arduino-Boards-like thing, with your forked plugins… you’ll have to fork Arduino-Boards too at the moment.

Kinda sucks, I agree, but I haven’t found anything better yet, either :frowning:

1 Like
7

I’m failing to picture how that directory structure works. You’re saying you essentially have a clone of Arduino-Boards in a subdirectory of a clone of Model01-Firmware (or equivalent), and you replace any submodule that you edit with your own fork, then you use your own script to update everything? Do you need to set up symlinks in order to do your builds? Am I even close to understanding how you manage your code? =)

8

No, I keep the submodules intact. I add a git remote, like: git remote add algernon git@github.com:algernon/Kaleidoscope. This allows me to have both upstream branches and my own branches in the same directory, and I can check out whichever I want. When I’m working on a plugin, I’ll check out my own branch, otherwise upstream’s master. I don’t have two copies, I have both the original and my fork in the same dir.

This means I have to be careful when I’m committing to Arduino-Boards, but I do that so rarely that a little extra care at those times is no big deal.

2 Likes
9

Aha! That’s conceptually tricky, but I’m pretty sure I could get comfortable with it. I’ll give it a try sometime soon.

1 Like
10

That’s what I’ve been doing as well; it’s nice because you can just checkout different branches to switch between your development version and the mainline version.

11

Can someone explain this with small words that make sense to a git novice? Getting people a git foundation that will allow them to work and potentially make contributions is on my list to document, but I don’t have the git-foo to write it without some coaching. While it’s an active conversation I’d like to capture at least some notes so I can follow up when I get down to this task on my list. :slight_smile:

12

References:
https://git-scm.com/book/en/v2/Git-Basics-Working-with-Remotes
https://git-scm.com/book/id/v2/Git-Branching-Remote-Branches

Most Git users are used to working with commands like push and pull (and maybe fetch) which interact with a server - called a remote in Git. Commonly, this remote is called origin.

Inside a single Git repo, you can have local branches and remote branches. Local branches are the ones that show up when you run git branch. Some of these branches (like master) are probably also on the server, but you can have local branches that you haven’t ever pushed to the server. Likewise, the server might have branches that you haven’t made local copies of.

The branches in the server repo are called “remote branches”, and you can see these with git branch -r. They will begin with the name of the remote, e.g. you will probably have at minimum origin/master.

You can checkout remote branches just like you can local branches - e.g. git checkout origin/master. This will bring you to what the server thinks is master. However, you can’t directly commit to remote branches. Instead, you can make a local copy of the remote branch with git checkout --track remotename/branchname to make a local branch branchname tracking the remote branch. Then you can push (or PR) that branch to the remote branch if you want to have your changes reflected on the remote branch.

However, there’s no reason you can’t have more than one remote. In this case, you could set origin to be your personal fork, and make a new remote upstream that contains the official repo. To do this:
git remote set-url origin git@github.com:myusername/myreponame
git remote add upstream https://github.com/keyboardio/reponame
git fetch --all
This last command gets the latest info from all your remotes.

Now you can interact with the upstream remote just like the origin remote. In particular, you can checkout upstream branches with git checkout upstream/branchname. You can make local copies of upstream branches. Etc.

You can even add more remotes. If I wanted to check out the awesome work @algernon is doing in his personal fork of something, just git remote add algernon https://github.com/algernon/reponame and git fetch algernon. Then I can check out algernon/master or any other branch that algernon has published for that repo.

Each local branch can be (optionally) configured as a “remote tracking branch” which means it’s designed to be (roughly) in sync with a particular branch on a particular remote. Your master branch is probably set up as a remote tracking branch for origin/master. For remote tracking branches, you can use git push and git pull without any other arguments, and Git knows which server and which branch to interact with.

You can set up branches to be remote tracking branches either for any remote you want. git checkout --track remotename/branchname does this for existing remote branches. Alternately, if you have an existing local branch you want to make a remote branch for (i.e. push to the public repo), you can use git push -u remotename branchname. After setting a branch up as a remote tracking branch, git push and git pull will work on that branch too, without any further arguments.

3 Likes
13

Thanks! bookmarking to get posted.

14

If nobody else has done this by the middle of next week (when I next have time to devote to it), I will get you a sequence of commands that accomplishes what @algernon described, with annotations.

Step by step instructions for each of several procedures are needed:

  1. Checking out the big repository tree (Arduino-Boards & dependencies)

  2. Building the firmware (from the command line) from that tree.

  3. Forking a module to work on, and linking the local copy to it.

  4. Creating a topic branch in that module, checking it out, making changes, and submitting a PR.

  5. Switching between your own branches and the keyboardio master.

  6. Cleaning up unnecessary branches, and other routine maintenance.

4 Likes
15

Just a head’s-up: I’m starting to work on this now. I might not have anything written down until early next week, though. Right now I’m experimenting a bit.

2 Likes
16

Thank you, Michael.

I just made a clean spot in the intro document adding the bit about how layers work, and am now going through a major revision to it. I realized there was a lot of stuff that was simply missing that would be very helpful, so I salute you from documentation land!

I would be thrilled to work through whatever you end up with and offer feedback. Just ask. :slight_smile:

17

Hey Michael, I created a template for repo-trackable firmware sketches that allows for easy downloading and updating of dependencies (even non-core/external ones), building the sketch firmware and flashing it to hardware. It’s most probably less comprehensive than what you want to end up with, but would you like to have a look? Maybe it’s a start, assuming that you aren’t finished, yet.

18

I’d love to see whatever you’ve got. I’ve been playing around with a few different methods anyway, so right now is the perfect time to try other ones.

2 Likes
19

Great! What’s the best way to get in touch with you outside the forum? Or just forum PMs?

1 Like
20

PM’s on the forum are pretty good; at the moment I actually look here more often than at my email.

2 Likes