Kubernetes Python Client Error Create_namespaced_binding: (409) Reason: Conflict

I'm using k8 v1.7 and Python Client v2.0. My custom scheduler detects a pending pod and schedules it successfully. However, after assigning a pod to a node, it complains that the p

Solution 1:

When a pod is created, the scheduler gets three "Pending" events:

1. Pod is not scheduled yet ('node_name': None, 'status': {'conditions': None,...})
2. Pod is scheduled ('node_name': 'some_node_name','status': {'conditions': [...,'status': True, 'type':'PodScheduled'],...} )
3. Pod is initialized but not ready yet ('node_name': 'minikube','status': {'conditions': [...,'status': True, 'type':'Initialized'], ... ,'status': False, 'type':'Ready']} )

Therefore, your custom scheduler should bind pod to the node on the first event, check the status of the pod and make sure it is scheduled when the second event appears, and then check if the pod is initialized when the third event appears.

If something goes wrong, the scheduler may need to take into account the previous errors and probably try to schedule the pod to different nodes.

In your case, your scheduler threats all three events like the first one and tries to schedule the pod again and again. That's why you see that "pod xxx is already assigned to node yyy" error.

Solution 2:

Here is the updated main method (according to @VAS's answer, thanks) to find the right PENDING pod that has not been scheduled yet.

defmain():
    w = watch.Watch()
    for event in w.stream(v1.list_namespaced_pod, 'default'): # default == namespace name# All pending pods have 3 states (not scheduled, scheduled, initialized but not ready yet)# We look for NOT SCHEDULED pod and conditions==Noneif event['object'].status.phase == 'Pending'and event['object'].status.conditions == Noneand event['object'].spec.scheduler_name == CUSTOM_SCHEDULER_NAME:
            print"Pending and Not Scheduled POD Found "+event['object'].metadata.name
            try:
                res = scheduler(event['object'].metadata.name,random.choice(nodes_available())) # nodes_available() returns all available nodesprint"success"except Exception as a:
                print ("Exception when calling CoreV1Api->create_namespaced_binding: %s\n" % a)