Lectures

i4 ComSys, Wehrle (SoSe)

Unix shell and programming language C
Process management: processes, threads, inter-process communication
CPU scheduling
Process synchronization, deadlocks
Memory management: virtual memory, segmentation, paging, fragmentation
File system and I/O system
Communication subsystem and sockets

from RWTH Online:

Aufgaben und Struktur von Betriebssystemen
Das Betriebssystem Unix
Systemaufrufe und Shellprogrammierung
Einführung in die Programmiersprache C
Prozessverwaltung: Prozesse, Threads und Interprozesskommunikation
Prozess-Synchronisation, Nebenläufigkeit und Deadlocks
CPU-Scheduling
Speicherverwaltung: Segmentierung, Paging, Fragmentierung, virtueller Speicher
Stack- und Heap-Verwaltung, Garbage Collection
Dateisystem und Rechteverwaltung
I/O-System
Verteilte Systeme
Socket-Programmierung

Power System Automation, Lankes

6SWS

Die Vorlesung vermittelt den Entwurf und den Aufbau von modernen Betriebssystemen. Hierzu wird zunächst der Aufbau von Rechnerarchitekturen analysiert und die Konsequenzen für die Systemsoftware gezogen. Insbesondere werden Sicherheitsrisiken erläutert und wie die Systemsoftware den Nutzer vor Angriffen schützt. Neben Sicherheitsaspekten vermittelt die Vorlesung, welche Möglichkeiten zur Leistungssteigerung existieren und wie Optimierungsziele der Ressourcenverwaltung erreicht werden.

Aufgaben und Struktur von Betriebssystemen
Einführung in UNIX
Programmierwerkzeuge
- Verwaltung von Source Code
- Einführung in dem Prozess des „Code Reviews“ und des automatisierten Testens
- Rust vs. C/C++
Systemnahe Programmierung
- Prozessormodelle (insbesondere Intel 80x86)
- Register- und Speichermodelle, Befehlsformate, Datentypen, Adressierungsarten
- Instruktionslevelparallelität / Datenparalleliät
- Mehrkern- und Mehrprozessor-Architekturen
  - Einführung in die nebenläufige Programmierung
  - Einführung in die Parallelisierung
  - Leistungsevaluation, Leistungsgrenzen (u.a. Amdahls Gesetz)
- Funktionsaufrufe und ihre Konventionen
  - Sicherheitsrisiken, mögliche Angriffsszenarien („Exploits“) und existierende Schutztechniken
Assembler und Assemblierung: Aufgaben, Funktionsweise
Laden, Binden:
- Statisches / dynamisches Binden, Laden mit / ohne Speicherverwaltung
- Angriffsszenarien und existierende Schutztechniken
  - Adress Space Layout Randomization (ASLR)
Entwurfsprinzipien
- Monolithische Betriebssysteme
- Microkernels (insbesondere seL4)
- Unikernels / Library OS
- Exokernels
- Funktionale Sicherheit der Systemsoftware
  - z.B. durch die Verwendung einer sicheren, systemnahen Programmiersprache
Betriebsmittel- und Prozessverwaltung
- Aufgaben der Betriebsmittel- und Prozessverwaltung
- Scheduling / Prozessplanung
- Prozesssynchronisation
- Verklemmungen
- Interprozesskommunikation
Unterbrechungen
- Arten und Aufgaben von Unterbrechungen
- Interruptsystem insbesondere des x86
- Systemaufrufe
Arbeitsspeicherverwaltung
- Paging und Segmentierung
- Demand Paging und Verdrängungsstrategien
- Seitenkanalattacken am Beispiel von Meltdown
Ein-/ Ausgabe
- Memory-mapped I/O
- Schichtung der E/A-Software
Dateisysteme
- Dateizugriff / Dateioperationen
- Struktur und Schichtung
- Journaling
Isolierung von Systemkomponenten
- Sandboxing
- Betriebssystemvirtualisierung / Container
- Virtuelle Maschinen

3SWS

Modelle, Grundlagen und Konzepte von Betriebssystemen (Linux/UNIX, Windows; Beispielarchitektur: 80x86).
Einleitung und Steuersprachen: Begriffsdefinitionen, Aufgaben und Struktur von Betriebssystemen, Steuersprachen und Shellprogrammierung (Bsp.: die Bourne-Shell), Programmierwerkzeuge.
Betriebsmittel- und Prozessverwaltung: Aufgaben der Betriebsmittelverwaltung, Prozessverwaltung, Prozesssynchronisation, Verklemmungen.
Unterbrechungen: Arten und Aufgaben von Unterbrechungen, das Interruptsystem der 80x86-Architektur.
Arbeitsspeicherverwaltung: Grundlegende Strategien der Speicherverwaltung, Seitenadressierung, Seitenwechsel auf Abruf, Seitenverdrängungsstrategien, Segmentierung, Segmentierung/Seitenadressierung/Zugriffschutz in der 80x86-Architektur.
Ein-/ Ausgabe: E/A in der 80x86-Architektur, Plattenspeicherverwaltung, Schichtung der E/A-Software.
Dateisysteme: Definitionen, Dateizugriff, Dateioperationen, Struktur, Schichtung, Beispiel: Aufbau eines I-Node-basierten Dateisystems

man pages

see book appendix F.8 “Documentation”
man man includes a list of all sections in the man pages
man -k some_keyword to search man pages for some_keyword
man -s 6 --regex -k '.*' to show the title of all manuals in section 6
standard sections:
- RETURN VALUES tells you what the function will return under success or failure
- ERRORS tells what happens under failure
- SYNOPSIS:
  - what arguments to pass to a library call,
  - what header files need to be included to use a library call
  - what libraries you need to link with to use the call
look for the definition of the structure timeval:
- using grep: use e.g. grep 'struct timeval' /usr/include/sys/*.h to look for the definition of the structure timeval in all files that end with .h in /usr/include/sys
- using the compiler: gcc -E main.c (where main.c includes the header time.h) will produce a C file that has all of the needed structures and prototypes in it, including the definition of the timeval struct
What is in section 6 “Games”?
- see ls /usr/games/
man -a kill: cycle through all of the different man pages named “kill”
- kill(2) means kill in section 2

Swap

From baeldung:

The swap space is located on disk, in the form of a partition or a file.
Linux uses it to extend the memory available to processes, storing infrequently used pages there.

Swappiness

From redhat

Occasionally, a system uses a high percentage of swap memory even when there is RAM available for use. The culprit here is the “swappiness” of the system.
The default value for swappiness is 60; however, you can manually set it anywhere between 0-100.
- Small values cause little swapping to occur, whereas high values can cause very aggressive swapping.
- A value of zero causes no swapping at all to occur, so if you want to minimize swapping to its lowest possible value without turning it off, you should set it to at least one.

Clear the Swap

From redhat

If Swap full, clear the swap by running:

Check space: # free -m 
Disable swap: # swapoff -a 

Wait approx 30 sec 
(use free -m to see the amount of swap used/available decrease over time)

Enable swap: # swapon -a 

top, htop

command	description
`man top`	e.g. search `res` to find the meaning of the RES column
/ (or F3)	search a process
t (or F5)	tree (shows eg parent processes → handy when you have a Zombie), press `-` or `+` to collapse and expand single item (collapsing all items only possible in newer `htop` versions using asterisk `*`)
k	signals (`SIGKILL`, `SIGTERM`, etc)
m	sort by memory usage
p	sort by processor usage
ctrl - f	scroll right
ctrl - b	scroll left

Memory

SHR vs RES vs VIRT (in top or htop command): askubuntu

VIRT

man top:

VIRT - everything in-use and/or reserved (all quadrants)

45. VIRT  --  Virtual Memory Size (KiB)
    The total amount of virtual memory used by the task.   It  includes
    all  code,  data  and  shared  libraries  plus pages that have been
    swapped out and pages that have been mapped but not used.

RES

man top:

RES  - anything occupying physical memory

22. RES  --  Resident Memory Size (KiB)
    A subset of the virtual address space (VIRT) representing the  non-
    swapped  physical memory a task is currently using.

%MEM

man top:

%MEM - simply RES divided by total physical memory

Shared Memory (SHR)

man top:

SHR  - subset of RES

superuser

The Linux Kernel: Memory Management:

Shared Virtual Memory

Although virtual memory allows processes to have separate (virtual) address spaces, there are times when you need processes to share memory. For example there could be several processes in the system running the bash command shell. Rather than have several copies of bash, one in each processes virtual address space, it is better to have only one copy in physical memory and all of the processes running bash share it. Dynamic libraries are another common example of executing code shared between several processes.

Shared memory can also be used as an Inter Process Communication (IPC) mechanism, with two or more processes exchanging information via memory common to all of them. Linux supports the Unix TM System V shared memory IPC.

t: SHR  --  Shared Mem size (kb)
   The amount of shared memory used by a task. It simply reflects mem‐
   ory that could be potentially shared with other processes.

S (htop only)

man htop:

STATE (S)
    The state of the process:
       S for sleeping (idle)
       R for running
       D for disk sleep (uninterruptible)
       Z for zombie (waiting for parent to read its exit status)
       T for traced or suspended (e.g by SIGTSTP)
       W for paging

Zombie

stackoverflow:

“You may sometimes see entries marked Z (or H under Linux, I don’t know what the distinction is) in the ps or top output. These are technically not processes, they are zombie processes, which are nothing more than an entry in the process table, kept around so that the parent process can be notified of the death of its child. They will go away when the parent process pays attention (or dies).”

Virtualization

Processes

process (abstraction) := a running program
a typical system runs hundreds of processes at the same time

Virtualizing the CPU

time sharing of the CPU: the OS creates the illusion that many CPUs exist by running one process, stopping it and running another etc.
- a resource (the CPU) is “divided in time” among the processes
- allows users to run as many concurrent processes as they want
drawback: if the CPUs are shared, each process will run more slowly
implementation via …
- mechanisms
  - low-level methods or protocols that implement a needed piece of functionality
    - e.g. context switch (“stop one program and start another on a given CPU”) (a “time-sharing mechanism”)
- policies
  - algorithms for making some kind of decision within the OS
    - e.g. scheduling policy (“decide which program the OS should run”)
- separation of mechanisms and policies
  - allows one to change mechanisms without having to change the policies and vice versa
  - a form of modularity

Machine State of a process

What constitutes a process?
What parts of the machine matter to the program execution?
machine state: what a program can read or update during its execution
- components:
  - address space
    - instructions
    - data (which the program reads and writes)
  - registers
    - program counter (PC) aka instruction pointer (IP)
    - stack pointer
    - frame pointer
  - I/O information (e.g. list of the files the process currently has open)
address space, registers

Process API (Overview)

must be included in any interface of an OS:
- create
- destroy
- wait
- suspend a process
- resume a process
- get status information, e.g.
  - how long a process has run for
  - what state a process is in
more details under Process API

Process Creation

loading of code and static data into address space
- eagerly
- lazily
allocate memory for the program’s stack
initialize the stack with arguments
- fill in parameters to the main() function (i.e. the argc and argv arrays)
allocate some memory for the program’s heap
initialization tasks related to I/O
- UNIX default for each process: initialize three file descriptors: stdin, stdout, stderr
start program at entry point main()
- OS transfers control of the CPU to the newly-created process

Process States

Running
Ready
Blocked = “not ready to run until some event (e.g. I/O completion, reading from a disk, waiting for a packet from a network) takes place”

Other states

initial state
- when process is being created
final state (aka zombie state)
- process exited, but has not been cleaned up
- allows the parent that created the process to examine the return code of the child and then call wait() to clean up the child

Scheduling

scheduled = “moved from ready to running”
descheduled = “moved from running to ready”
OS scheduler = e.g. decides …
- to run process 2 while process 1 issued an I/O,
- not to switch back to process 1 when I/O is completed

Data structures

process list (aka task list)
process control block (PCB)
- register context (for context switches)

Process API

File Descriptors

from File Descriptor:
- On Linux, the set of file descriptors open in a process can be accessed under the path /proc/PID/fd/, where PID is the process identifier. File descriptor /proc/PID/fd/0 is stdin, /proc/PID/fd/1 is stdout, and /proc/PID/fd/2 is stderr. As a shortcut to these, any running process can also access its own file descriptors through the folders /proc/self/fd and /dev/fd.
  UNIX System calls

strace

overview: thegeekstuff:
strace some_command: trace system calls and signals
strace ./avail_mem_malloc -l: trace system calls and signals when running the command ./avail_mem_malloc -l (source code of ./avail_mem_malloc -l is in /home/bra-ket/Desktop/Aachen/Uni/Basics_CS/OS/git/C_exercises/day_20_memory/avail_mem_malloc.c)
strace -e mmap ./avail_mem_malloc -l: filter a specific system call and/or signals, here: the mmap system call

mmap, munmap: map or unmap files or devices into memory
brk, sbrk: change data segment size

Resource Limits

setrlimit(), getrlimit()

see example setrlimit.c
see man setrlimit
- Trick: to get man pages in color:
  - install the most pager using sudo apt-get install most
  - in terminal set export PAGER="most"
  - to change most keybindings to vim keybindings cp -iv /usr/share/doc/most/lesskeys.rc ~/.mostrc

ulimit

get and set user limits
- i.e. not for one process, but for one user!
see man ulimit
current user limits: ulimit -a

Counting Processes

see how-to-count-the-number-of-processes-in-linux
ps aux
- aux are BSD style flags
  - a = show processes for all users
  - u = display the process’s user/owner
  - x = also show processes not attached to a terminal

Visualize Memory Usage

smem

see How to visualize memory usage on Linux
to fix the NameError: name 'xrange' is not defined edit /bin/smem and apply this patch

sudo apt install smem

# Pie chart
sudo python3 /bin/smem --pie name -c "pss" -U some_username

# Bar chart
sudo python3 /bin/smem --bar name -c "pss uss" -U some_username

Concurrency

Multi-Threaded Programs

gcc -pthread vs -lpthread

from Difference between -pthread and -lpthread while compiling
- -pthread tells the compiler to link in the pthread library as well as configure the compilation for threads.
- Using the -lpthread option only causes the pthread library to be linked - the pre-defined macros don’t get defined.
- Bottom line: you should use the -pthread option.

Share on

Twitter Facebook LinkedIn

Operating Systems Notes

Lectures

i4 ComSys, Wehrle (SoSe)

Power System Automation, Lankes

6SWS

3SWS

man pages

Swap

Swappiness

Clear the Swap

top, htop

Memory

VIRT

RES

%MEM

Shared Memory (SHR)

S (htop only)

Zombie

Virtualization

Processes

Virtualizing the CPU

Machine State of a process

Process API (Overview)

Process Creation

Process States

Other states

Scheduling

Data structures

Process API

File Descriptors

UNIX System calls

strace

Related to Memory

Resource Limits

setrlimit(), getrlimit()

ulimit

Counting Processes

Visualize Memory Usage

smem

Concurrency

Multi-Threaded Programs

gcc -pthread vs -lpthread

Share on