Maximilian Stark (mail@dakror.de), WS2019

Stand: 22.01.2019

Advanced Topics of Software Engineering

Context of Software Engineering

Characteristics of Software Systems in different domains

• Reactive systems
  • Properties
    • Interactivity
    • Nonterminating processes
    • Interrupt-driven
    • State-dependent response
    • Environment-oriented response
    • Parallel processes
    • Stringent real-time requirements
  • Embedded Systems
    • Properties
      • Processing
      • Communication
      • Storage
      • User Interface
    • Quality attributes
      • Dependability
      • Efficiency
      • Real-time usage
  • Real-time systems (RTS)
    • Deadline dependency
    • Importance of predictability
    • Processing of Events
• Information systems
  • Collection and delivery of information and knowledge
  • Data intensive platforms
  • Business internal systems (ERP, etc...)
  • Quality attributes
    • Dependability
    • Scalability
    • Stability
• Cyber-physical systems
  • European view: Socio-technical systems, integrating IS, ES & humans
  • US view: New term for ES

From requirements to system design

Software Architecture

• Efficiency
• Risk minimization
• Communication means
• Knowledge management

Model of Software Architecture

• Mission
• Systems
• Environment
• Architecture
• Architectural Description
• Rationale
• Stakeholder
• Concerns
• Viewpoints
  • 4+1 view model
    • Logical view: end-user & designers
    • Implementation view: programmers
    • Deployment view: system engineers
    • Process view: system integrators
    • Use-case view
• Views
• Models
• Library Viewpoints

Models

• Goal: Abstraction
• Types
  • Object model
  • Functional model
  • Dynamic model
  • Data model
• Attributes
  • Reduction
  • Pragmatism
  • Mapping
• Modeling process
  1. Understand
  2. Conceptualize
  3. Abstract
  4. Define
  5. Construct
  6. Evaluate
  7. Refine
  8. repeat

Modularity

• Unit of measurement for separability of components
  • Coupling
  • Cohesion
• System decomposition
• Goals
  • Information hiding
  • Understandability increase
  • Work distribution
  • Complexity reduction (divide and conquer)
    • Functional decomposition
    • Modular decomposition
• Types of decomposition
  • black-box model: functional (de-)composition
  • white-box model: constructional (de-)composition

Component-based software engineering (CBSE)

• Component characteristics
  • Interaction through public interfaces
  • Independent executable entities
  • Standardized
  • Documented
  • Composable
  • Deployable
• Component model
  • Interface
  • Usage information
  • Deployment and use
• Contras
  • Questionable trustworthiness
  • Certification
  • Emergent property prediction
  • Requirement trade-offs
• Microservices
  • Independently deployed components
  • Language-agnostic communication (REST)

Design by Contract (DbC)

• Dependable and robust OOP software
• Increased code reuse
• Contract between Client and Supplier
• Core principles
  • Non-redudancy: No testing of own precondition
  • Reasonable preconditions
  • Failure model
  • Disciplined exception handling
    • Retry
    • Failure
  • Exception sinplicity
• Specifications of ubiquitous interfaces
  • Preconditions
  • Postconditions
  • Invariants
  • Side effects
• In practice: assert

Defensive programming

• Strong verification of system inputs and state
• More work than DbC
• Many redundant checks

Dependency structure matrix (DSM)

• Column depends on row
• Partitioning
  • Transformation into block triangular form
  • Collapsing of mutually dependent components into composite entity
  • Elimination of cycles

Guidelines for modular design

• Low coupling, high cohesion
• Single responsibility principle
  • Never more than one reason to change a class
  • Achieve cohesion at package level
• Separation of concerns
  • Minimize responsibilities-per-component ratio
  • Functional cohesion
• Open/closed principle
  • Open for extensions
  • Closed for modification
• Liskov substitution principle
  • Subtypes must be fully substitutable for base type
• Interface-segregation principle
  • No forced dependency on unneeded interfaces
• Change anticipation
• Limited repetition
  • Types of duplication
    • Imposed
    • Inadvertent
    • Impatient
    • Inter-developer

Coupling and Cohesion

• Coupling
  • Quantification of external dependencies
  • Types from low to high
    1. Data: Pure data passing
    2. Stamp: Complete data structure transmission, sharing of data format
    3. Control: Absolute dependency on outputs of other component
    4. External: I/O
    5. Common: shared data between components
    6. Content: duplicate content with need to cross-update across components
• Cohesion
  • Quantification of internal interdependencies
  • Types from low to high
    1. Coincidental
    2. Logical
    3. Temporal
    4. Procedural
    5. Communicational
    6. Sequential
    7. Functional
  • Hard to quantify
    • Analysis of description in natural language
    • LCOM (Lack of Cohesion in Methods)
      • Number of method pairs that do not use the same attributes
      • Good values indicate good design, bad values are meaningless

Formal definition of system structure

• Systems as tuple $S:= (C,I,CON)$
• Components $C$
• Interfaces $I$
• Environment $env \in C$
• Connections $CON \subseteq I \ times I$
• Functions
  • $\mathrm{parent}(c): C \mapsto C$: Parent of component or $env$ for root
  • $\mathrm{assigned}(i): I \mapsto C$: Component for given interface
  • $\mathrm{connected}(c): CON \mapsto (I \times I)$: Connection between interfaces

Quality

• Internal
  • Modularity
  • Testability
• External
  • Attack surface
  • Performance
• Intrinsic
  • Correctness
  • Completeness
• Design-time
  • Portability
  • Maintainability
• Run-time
  • Usability
  • Reliability
  • Efficiency
  • Functionality

Antipatterns

• Causes: The 7 deadly sins of software
  1. Haste
  2. Apathy
  3. Narrow-mindedness
  4. Sloth
  5. Avarice
  6. Ignorance
  7. Pride (Hubris)
• Blob: Single magic universal class doing everything
• Functional decomposition: Lack of OOP
• Auto-generated stovepipe: Ad-hoc repairs of ever-growing legacy support
• Golden Hammer: One tool for everything, even if bad choice
• Design by Committee

Reuse

• Challenges / Issues
  • Technical issues
  • Organizational issues
  • Conflict between flexibility and stability
• Principles
  • Modularity
  • Loose coupling & high cohesion
  • Information hiding
  • Separation of concerns
• Types
  • Opportunistic / Ad-hoc
    • Not scalable
    • Clone-and-own paradigm
    • Lack of structure
    • Clone detection
  • Planned / Structured
    • Software product lines
      • Software construction kits with stable core architecture
      • Managed variability modeling
    • (Software) Frameworks
      • Dedicated library projects for 3rd site use
• Variability
  • External: Different artifacts for user / customer
  • Internal: Different internal components
  • Managed Variability
    • Definition of variability at domain engineering
    • Exploitation of variability in application engineering
    • Moment of variability resolution: "Binding time"
• Orthogonal variability modeling
  • Variability of a software product line in regards to other software development models

Software Product Line Engineering (PLE)

• Proactive development for reuse
• "Family Engineering" - Design of products with mass variability options
• Specializations
  • Platform
  • Environment
  • Functionality
  • Process
• Process
  1. Requirements on $n$ individual systems
  2. Family engineering: Information integration
  3. Product line infrastructure
  4. Application engineering: Information specialization
  5. $m$ resulting individual systems
• Scoping
  • Aspects
    • Products
    • Domains
    • Features
    • Assets
  • Process
    1. Identify
    2. Prioritize
    3. Optimize

Reference Architecture

• Abstract software architecture for specific application area
• Definition of software elements and their interactions
• =Blueprints
• Types
  • Functional: Structuring required functionality
  • Logical: Layer and component definitions
  • Technical: Tech-stack definitions
• Examples
  • Database-centric: Mainframes
  • Message-oriented: RMI
  • Object-oriented: .NET
  • Service-oriented: WSDL

Testability

• Conflict with low time-to-market
• General testability scenario
  1. Source
  2. Stimulus
  3. Artifact / Environment
  4. Response
  5. Response measure
• Definitions in the literature
  • Facilitation of testing (efficiency)
  • Facilitation of revealing faults (effectiveness)
    • RIPR Model: Requirements for failure reveal
      • Reachability: Fault can be reached
      • Infection: Fault corrupts program state
      • Propagation: Faulty state gets carried to corrupt final state
      • Revealability: Final state contains faults
  • Other focus areas (requirements testability)
  • Measurement of effort (unproven)
• Properties of programs affecting testability in order of significance
  1. Observability
  2. Controllability
  3. Complexity / Simplicity
  4. Isolateability (Coupling & Cohesion)
• Assessment methods
  • Theoretical
    • Measurement of information loss
      • Implicit: Same result when given different inputs
      • Explicit: Loss of some calculated values by design
    • Domain/range ratio (DRR)
      • Porr testability if $|input domain| >> |output domain|$
      • Black-box testability
      • Relating to single function domain
      • Correlation with implicit loss
  • Mutation-based
    • Injection of defect into program
    • Execution of random testing
    • Measurement of likelihood of output change due to mutation
    • High likelihood = High testability

Safety

• Freedom of risk
• System failure types
  • Cascading failure
  • Common cause failure (CCF)
• Risk reduction
  • Avoidance of systematic faults during design
  • Tolerance of systematic faults at runtime
  • Tolerance of random faults at runtime
• Functional safety
  • hazards & risks purely through operation
  • Lifecycle approach
    • Integration of safety into design
    • Systematic hazard idenfitication & analysis
    • Protection layers
    • ISO 26202 Safety lifecycle model
      1. Concept Phase
         1. Item definition → External measures
         2. Initiation of safety lifecycle
         3. Hazard & risk assessment → Controllability
         4. Functional safety concept → Allocation to other technologies
      2. Development Phase
         1. Product development & Safety validation
         2. Production Planning
         3. Operation Planning
         4. Functional safety assessment
         5. Release for production
      3. Production / Operation
         1. Production
         2. Operation
• Safety analysis
  • Qualitative
    • Identification of failures
    • Knowledge about relevant fault types required
    • Example: HAZOP
  • Quantitative: Fault frequency predictions
  • Inductive Analysis Methods
    • Bottom-up (from causes to effects)
    • Failure Mode and Effects Analysis (FMEA)
  • Deductive analysis Methods: Top-down
    • Fault Tree Analysis (FTA)

Failure Mode and Effects Analysis (FMEA)

• Uncommon in software engineering
• System-level analysis
• Establishing (software) test goals
• Table columns per fault
  • Failure mode
  • Failure effects
  • Causes
  • Detection / Prevention measures
  • Severity (SEV) (1-10=hazardous without warning)
  • Occurrence (OCC) (1-10=persistent)
  • Detection (DET) (1-10=unlikely)
  • Risk priority number (RPN) = SEV OCC DET

Fault Tree Analysis (FTA)

• Goal: Identification of system failure conditions
• Graphical representation of causal relationships
• Done when design / architecture is available
• Requirement of system understanding, after FMEA
• Reversed attack tree
• Procedure
  1. System plan as input
  2. System definition under scrutiny (boundaries)
  3. Definition of undesired top event
  4. Recursive identification of series of events leading up to top level event
  5. Cut set identification: Set of events leading to failure
• Events
  • Primary events
    • Basic events: No precursor, probabilistic
    • Undeveloped events: No major effect on the system
    • External events: Expected events, not faults
  • Intermediate events: AND/OR gate link
  • Expanded events: Reference to whole other subtree

Model-Based Design

• Advantages
  • Separation of concerns
  • Model in the Loop (Simulation)
  • Understandability
  • Reusability
  • Code-Generation
  • Analysis: Verification and Validation
• Components
  • Operational context
  • Interfaces
  • Behavior
  • Inner structure

Model consistency

• Consistent-by-construction: One metamodel of entire system
• Consistent-by-check: Agreement of consistency between different view-types across tools

ISO 26262 Component model

• Components
  • Item: System(s) under consideration
  • System implements function(s)
  • Systems are (composed of) elements
  • Non-atomic elements are components
  • Atomic elements are software parts / hardware units
• Properties
  • Modular construction
  • Generic levels of abstraction
  • Inter-level allocation
  • Distributed development
  • Reusable elements

Modeling Viewpoints

• Requirements
• Functional
• Logical
  • Reactive behavior
  • Computational behavior
  • Platform independent
• Technical
  • Hardware
  • Software
    • Dynamic: Runtime
    • Static: Design-time
  • Deployment logic model: Mapping logical to technical

Security

• Core properties: CIA
  • Confidentiality
  • Integrity
  • Availability
• Other desirable properties
  • Non-repudiation
  • Authenticity
  • Auditability
  • Accountability
  • Privacy
  • Anonymity
• Design principles
  • Least privilege
  • Complete mediation
  • Secure, fail-safe defaults, Permission refusal by default
  • Compartmentalization
  • Minimum exposure
• Trade-offs
  • Performance
  • Usability
  • Time to market
  • Cost objectives

Distributed System Architectures

Database-centric

Operational database

• Online transaction processing (OLTP)

Data warehouse (DW)

• Subject oriented
• Better data analysis tools
• data homogenization from varying internal sources
• Core business intelligence data storage
• Online analytical processing (OLAP)
• Extract, Transform, Load (ETL)
  • Procedure for storing multidimensional data in DW
  • Problems
    • Different sources
    • Different formats
    • Missing data / gaps
    • Query performance of DW
• Architectures
  • Central: Single DW
  • Federated
    • Data marts: distributed subdata nodes
    • Virtual DW
  • Tiered
    • Hierarchical storage in aggregated data cubes
    • Materialized central DW

Big data

• Central Vs
  • volume
  • velocity
  • variety
  • veracity
  • value
• Sharding strategies (subdata assignments and routing)
  • Lookup
  • Range
  • Hash
• Key Value stores
  • Redis
  • Various data types
  • In-RAM database
• Bigtable
  • Mapping of row, column key and timestamp to arbitrary byte array
  • Lexical sorting of rows
  • Column hierarchy notation
• Document-oriented
  • MongoDB
  • JSON values
• Graph-database
  • Core consistency rule: no broken links
  • Joins as first class citizen
  • Neo4J

Message-oriented

• Message intents
  • Command
  • Document
  • Event
• Industry Standard: Java Message service
  • pub sub with in/out queues
• Message queues
  • point-to-point pattern: 1:1 conversation per message
  • pub/sub
  • datatype-based
  • shared queues: local distribution
• Message broker challenges
  • Scalability
  • Security
  • Monitoring & debugging
  • Maintenance
• Reactive systems
  • Properties
    • Responsive
    • Resilient
    • Elastic
    • Message-driven
  • Actor model
    • Purely reactive modeling encompassing
      • processing
      • storage
      • communication
    • Possible actions upon message receive
      • Creation of new actors
      • Sending of messages to known actors
      • Designation of handling procedures for upcoming message

Object-oriented

Object broker

• Object oriented RPC
• CORBA: Common Object Request Broker Architecture
  • language agnostic standardized specification
• gRPC (+protobuf)

Component-based

Java EE

• Multi tier architecture
  • Client tier
  • Web tier
  • Business tier
  • Enterprise information system tier (ERP)
• Application server
  • System resource mediation
  • Life-cycle and deployment management
  • Communication and security management
  • Transaction and persistence management
  • Functionality encapsulated into containers
    • Application client container
    • EJB container
    • Web container
• Enterprise Java Beans (EJB)
  • RMI for interaction
  • Types
    • Session (transient)
      • Stateful
      • Stateless
      • Singleton
    • Message driven
    • Entities
  • Contents
    • Business logic implementation
    • Business interfaces
    • Helper classes
  • Client access
    • no-interface view: Exposure of public methods to (local) client
    • business interface
  • Client obtainment
    • Dependency injection
    • Java Naming and Directory Interface (JNDI) lookup

AUTOSAR

• Automotive Open System Architecture
• Standardized automotive reference architecture
• Focus on non-functional requirements
  • Modularity
  • Scalability
  • Portability
  • Reusability
• Architecture workflow
  1. Modular software-components (SW-C)
  2. Virtual functional bus (VFB): communication between SW-C
  3. System constraint and ECU descriptions
  4. Embedding on ECUs (engine control units)
• ECU stack
  • Runtime environment (RTE)
  • Module layers
    • Application layer
    • AUTOSAR RTE layer
    • Basic Software layer
    • Micro-controller abstraction layer
• Connection paradigms
  • Client/Server
  • Sender/Receiver a la pub/sub
• Standardized module interfaces

Android

• Architecture
  • Layers
    • Application
    • Framework
    • Library
    • Runtime
    • Kernel
  • Information hiding
  • high extensibility
  • Sandboxing of Apps into individual VMs: Zygote

OSGi

• Open services gateway initiative
• Specificationas for dynamic component systems
• Components
  • Bundles: actual developed components
  • Services: dynamic connection layer for bundles
  • Life-Cycle: bundle management
  • Modules: import/export declarations for bundles
  • Security
  • Execution environments
• Driving forces
  • Reduced complexity (information hiding)
  • Reuse
  • Real World (dynamic architecture)
  • Easy development
  • Dynamic updates
  • Adaptive
  • Transparency
  • Versioning
  • Simplicity
  • Small size
  • Speed
  • Laziness
  • Security
  • Humbleness
  • Non-intrusiveness
  • Runs everywhere
  • Widely used

Service-oriented

SOA

• Process orchestration of enterprise-level business services
• Solution of specific problems in concrete scenarios
  • Multiple languages
  • Multi-governance
    • Strategic level: management of company
    • Operational level: across departments
    • Technical level: Technical integrity
  • Interoperability via standard
  • Independence from vendor or technology
  • Loose coupling
• Drawbacks
  • Lacking ecosystem
  • Scalability
  • Unfunctional browsing
  • High complexity
  • Reduced agility
• Principal architecture
  • Application frontend
  • Service
    • Contract
    • Interface(s)
    • Implementation
  • Service repository
  • Service bus
• Roles
  • Service consumer
  • Service provider
  • Service registry
• Principles
  • Loose coupling
  • Modularization
  • Service contract
  • Discoverability
  • Abstraction and autonomy
  • Reusability
  • Composability
  • Statelessness
• Web services
  • SOAP
  • REST

SOAP

• XML based
• Technology agnostic
• Verbose (arguably bloated)
• Structure
  • Envelope
    • Header
      • Header block(s)
    • Body
      • Body block(s)
• Usage variations
  • Interaction styles
    • Document-style: exchange of hierarchical document data
    • RPC-style: parameterized call and response
  • Encoding styles
• Binding: definition of correct message-wrapping in a given transport protocol
• Web service description language (WSDL)
  • Specification of service interfaces
  • Difference from IDLs: Specification of required service access mechanism
  • Structure
    • Abstract definitions
      • Types
      • Messages
      • Operations: simple exchange of messages
      • Port types: logical collection of related operations
    • Concrete definitions
      • Bindings
      • Services & ports

Representational state transfer (REST)

• Stateless
• Cache-support
• Focused NFRs
  • Performance of component interaction
  • Ecosystem scalability
  • Interface simplicity
  • Visibility
  • Language-agnosticism
• Null-style
  • Empty set of constraints
  • Starting point for REST-description
  • Description of system without component boundaries
• Client-Server
  • Separation of UI from data storage
  • Increased portability
  • Improved scalability
  • Independent component evolution
• Uniform interfaces
  • Uniform identification scheme
    • URI
      • URL (location)
      • URN (name)
      • Structure
        • Scheme
        • Authority
        • Path
        • Query
        • Fragment
  • Uniform representation of exchanged information
  • Decoupling of implementations from the services they provide
  • Trade-off
  • Hypermedia as the engine of application state (HATEOAS)
  • MIME-Types: xml, json, etc
• Code-on-demand: Downloadable extension scripts

Microservices

• Conways law: Design of systems inevitably resembles team structure
• Products instead of projects: Full lifecycle responsibility of team
• Characteristics
  • Decentralized governance
  • Decentralized data management
    • Problem: significant temporal coupling
    • Varying world views among services
  • Infrastructure automation
  • Design for failure
  • Evolutionary design
• Patterns
  • Data management: database-per service / shared db
  • Communication: service discovery / API gateway
  • Deployment: service-per-container
  • Operation: circuit breaker, distributed tracing
  • UI: client side UI composition

From source code to deployment

• Version control
  • SVN
    • Centralized
    • Storage of file based diffs
  • Git
    • Distributed
    • Snapshots of miniature file system
• Continuous integration (CI)
  • Risk reduction
  • Quality control
  • Health reports
  • Availability of stable release
• Continuous deployment (CD)
  • Continuous delivery
    • Software engineering approach to ensure release-ready software at any point in time
    • Short development cycles
    • Benefits
      • Time to market
      • Customer satisfaction
      • Productivity & Efficiency
      • Building the right product
  • Immediate deployment to production
  • Deployment pipelines
    • Individual business strategies
    • Smoke-testing
    • Deploy to production-mirror
  • Deployment automation patterns
    • Repository: all relevant data under version control
    • Scripted deployment: full deployment process in scripts
    • Single command deployment: single invocation to deploy
    • Unified deployment: single script with support for multiple environments and platforms
    • Template verifier: Single template file as base for all target environment properties
    • Externalized configuration: extraction of application variables into build-time properties
    • Tokenize configuration: Replacement of certain values through externalized configuration
    • Remote deployment: Usage of centralized machine or cluster for deployment
  • Blue green deployment
    • Usage of production mirror
    • Final stages of testing after deployment on mirror
    • Flipping of routing for mirror and actual production
• Virtualization
  • VMs
  • Containers
• Cloud
  • Goals
    • Availability & Reliability
    • Reduced investments and proportional costs
    • Stability
  • Actors
    • Consumer
    • Provider
    • Auditor: independent cloud service evaluation party
    • Broker: negotiator between consumer and provider
    • Carrier: intermediary for transport and connectivity of consumer and provider
  • Characteristics
    • On-demand self-service
    • Broad network access
    • Resource pooling
    • Rapid elasticity
    • Measured service
  • Forms
    • IaaS
    • PaaS
    • SaaS
  • Deployment models
    • Public cloud
    • Private cloud
    • Outsourced private cloud
    • Community cloud
    • Hybrid
  • Architectures
    • Workload distribution
    • Resource pooling
    • Dynamic scalability
      • Horizontally
      • Vertically
      • Relocation
    • Service load balancing
      • cloud usage monitor
      • Resource cluster
      • Resource replication