Working Space for the Error Codes, Faults, Logging Enhancement Development Proposal

This page and any subpages are a jointly edited space to centralize materials the LCOO staff need to share with each other to prepare our Development Proposals and Joint Collaboration through related project teams. We create such collaboration spaces based on identification of similar interests and possibly related efforts.

Formal OpenStack Submissions/efforts/projects

Type	Short Description
	This Development Proposal seeks to correct platform-wide log/error/fault deficiencies which are a major pain point for operators and their tenants

Team members: (Add your name here to be included in activities and be notified as this page evolves)

Jill Sales, Andrew Ukasick, Uttam Dravidam, Hank Greene

Planned working sessions

Date	Joint Meeting Page x-ref	Coordinator
	Being planned, please add your availability here: (doodle needed)

Other Related Materials

x-ref	Materials or URL	Submitter (for more info)
	LCOO-5 - Getting issue details... STATUS	Andrew Ukasick
	Error Codes, Faults, Logging Enhancement Development Proposal	Andrew Ukasick
	Slack #logging channel: https://lcoo.slack.com/messages/C2CJCTX7U/	Andrew Ukasick

	Logging Working Group’s wiki page: https://wiki.openstack.org/wiki/LogWorkingGroup	Andrew Ukasick
	OpenStack General Logging Guidelines: https://specs.openstack.org/openstack/openstack-specs/specs/log-guidelines.html	Andrew Ukasick
	OpenStack Security Log Guidelines: https://wiki.openstack.org/wiki/Security/Guidelines/logging_guidelines	Andrew Ukasick
	Fault Genes working group’s wiki page: https://wiki.openstack.org/wiki/Fault_Genes_Working_Group Fault Genes Presentation: OpenStack_Fault_Management_Framework-Fault_Genes.pdf	Andrew Ukasick

Logging related Specs / Blueprints

SPEC-Oslo http://specs.openstack.org/openstack/oslo-specs/specs/pike/improving-logging-debugging.html
SPEC-Oslo-OSprofiler cross service & project profiling https://specs.openstack.org/openstack/oslo-specs/specs/mitaka/osprofiler-cross-service-project-profiling.html
BP-Nova-OSProfiler in Nova https://blueprints.launchpad.net/nova/+spec/osprofiler-support-in-nova
BP-Magnum-OSProfiler in Magnum https://blueprints.launchpad.net/magnum/+spec/osprofiler-support-in-magnum
BP-Manila-OSProfiler in Manila https://blueprints.launchpad.net/manila/+spec/manila-os-profiler
BP-Senlin-OSProfiler in Senlin https://blueprints.launchpad.net/senlin/+spec/senlin-osprofiler
BP-Horizon-OSProfiler in Horizon https://blueprints.launchpad.net/horizon/+spec/openstack-profiler-at-developer-dashboard
BP-OSProfiler Overhead control in OSProfiler https://blueprints.launchpad.net/osprofiler/+spec/osprofiler-overhead-control

Related Resources/Links

Here are some general logging related etherpads from recent community events:

Etherpad for logging sessions at the Austen Summit in April 2016: https://etherpad.openstack.org/p/AUS-ops-Logging
Etherpad for Logging sessions at the Ops Meetup in August 2016 : https://etherpad.openstack.org/p/NYC-ops-Logging-and-monitoring
Etherpad for Logging sessions at the Barcelona Summit October 2016: https://etherpad.openstack.org/p/BCN-ops-logging-monitoring
Etherpad for Logging sessions at the Ops Meetup in March 2017: https://etherpad.openstack.org/p/MIL-ops-logging-and-monitoring
Logging Working Group sessions at the Boston Summit May 2017: https://etherpad.openstack.org/p/BOS-forum-log-messages (Scroll to bottom for LOTS of other resource links)

Here are a few examples of previous attempts to address the problem:

A “Development Proposal” to address logging that was started and stalled: https://review.openstack.org/#/c/409878/6/user-stories/proposed/logging-error-codes.rst It appears that
A “Development Proposal” to address Traceability in logs that was started and also seems stalled: https://review.openstack.org/#/c/440309/9/development-proposals/proposed/traceability-in-services.rst
The most recent stab at addressing this that I’m aware of: https://etherpad.openstack.org/p/logging-better-debugging-proposal
Traceability work is in process. Sean Dague has been posting regular updates to the -dev mailing list about the work since the Boston Summit (see http://lists.openstack.org/pipermail/openstack-dev/2017-June/117924.html for a relatively recent example).
Log improvement thread of Log WG discussed in Boston: http://lists.openstack.org/pipermail/openstack-dev/2017-May/116619.html

A must read history and analysis of Logging efforts in the Community from 2010 to present: https://doughellmann.com/blog/2017/04/20/lessons-learned-from-working-on-large-scale-cross-project-initiatives-in-openstack/